We help IT Professionals succeed at work.

We've partnered with Certified Experts, Carl Webster and Richard Faulkner, to bring you a podcast all about Citrix Workspace, moving to the cloud, and analytics & intelligence. Episode 2 coming soon!Listen Now

x

VSAN Storage Scalability

Medium Priority
36 Views
Last Modified: 2020-05-20
VSAN Storage Scalability

If I understand VSAN is the technology that enables the sharing of each ESX host Disk to make a large shared disk.

For instance if you have ESX1 and ESX2 with 40 GB disk each, then you can create one large Shared disk of 80 GBs combining ESX1 and ESX2 disks.

Now I would like to know how scalable it is, in case we need a large amount of storage

.For instance,  dedicated storage such as EMC, they provide Racks of storage that can hold pitabytes of data.
So is VSAN used in environment where they do not have to much data to store and sometimes they will have to offload the old data off the VSAN to an external storage device?

Any Expert to elaborate in that point?

Thank you
Comment
Watch Question

Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
There are two ways to scale vSAN (generic) as there are multiple flavors unless you're focused on VMWare vSAN.

The first methodology is to increase the ESXi nodes and scale, this can be espensive as each ESXi node would require a license, hardware, etc..

The second methodology is to have hardware only nodes.  Depending on what brand of vSAN some come in storage only nodes so that you can scale accordingly. Normally you would have to purchase at least two of these for redundancy.

In my experience it has always been less expensive to scale with SAN hardware such as EMC, Netapp,Nimble, etc.. since you only only need to add shelves.  

For instance this is VMWare vSAN scalability Limits, essentially the max size of a single datatastore on ESXi host per host, 62TB and it keeps two copies of the VM on different hosts so you'll need to multiple that by 2X for actual capacity.

StarWind vSAN is a different product   https://www.starwindsoftware.com/starwind-virtual-san from what I've seen it is a less expensive solution that allows you to scale up and out but may not be an enterprise solution.  

My opinion is that to get to PB level you will still need to use hardware based SAN solutions.




andyaldersaggar maker's framemaker
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
A host's "internal" capacity is not limited to the physical internal disk bay count as far as vSAN or other technologies are concerned, you can add additional PCIe controllers to a server with external SAS ports and connect it to a rack full of disk enclosures. Logically they are direct attached [internal] even though they are in external cages. Biggest single server I built commercially had 208 disk bays - 8 internal disks plus 8 external shelves with 25 disks in each. Compaq made far bigger than that to break single server SQL benchmarks.
Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
We need to define what a vSAN consists of because there are essentially two different techonologies.

1.   HCI (Hyperconverged) - This type of vSAN distributes your workloads on multiple nodes in your VMWare environment
2.   Single host as Andy mentioned is a vSAN technology but is more akin to your generic physical SAN infrastructure.  Normally you have several nodes in a Supermicro chassis and PCI cards going to your shelves.  This technology has been in production for the last decade by vendors such as Lefthand, Nexenta,  Oracle ZFS Appliance,  and many other vendors  It is a solid technology and in many cases uses ZFS that is similar to OS to what Netapp provides.

Number 1 option is what allows you use ESXi hosts for compute and storage.  The number 2 option segregates your compute and Storage onto different hardware and is structured to be separate.  

andyaldersaggar maker's framemaker
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Paul, that a single host can have a couple of hundred disks attached does not preclude you having several of them similarly configured in a cluster. A single host is not a vSAN. I was merely showing that external shelves are not precluded from the HCI model.

Author

Commented:
The VSAN I am aware of is when you have several ESX hosts with internal hard drives, then you configure those internal hard drives as a one large shared hard drive.

if VSAN , The way you have described it above, can be just like regular SAN, the drives are in a separate rack not internal to ESX hosts, then how is that VSAN different from a regular SAN.?

 
andyaldersaggar maker's framemaker
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
A host with an external direct attached shelf is still a host, adding an external shelf is no different logically than adding an additional internal disk cage. If you wrapped sticky tape around the servers and their external shelves they would in effect just be servers with more internal bays than normal.

Whilst there's not many VMware vSAN external HBAs VMware are not the only maker of vSANs, you could use StarWind for example.

Author

Commented:
So the difference between SAN and VSAN is the Adapters that connect ESX Host to storage.
SAN uses Fiber channel, ISCSI, NFS, FoE

VSAN uses different Tech ?
andyaldersaggar maker's framemaker
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
vSAN uses local disks so generally SAS.

Also please note it is vSAN and not VSAN.  which is a FC protocol.
Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
The question was regarding scalability.  In general hyperconverged vSAN options will be limited due to the architecture.  software based SAN (keep in mind vSAN is loosely used in similar fashion as "cloud")  and hardware based SAN options have better ability to scale up and out (adding more shelves versus adding ESXi host nodes in addition to architecures that allow you to add more SAN compute nodes such as Netapp cluster mode or EMC Isilon)

What type of infrastructure are you looking to deploy?  Once you define requirements and scope you can look for a solution to meet the requriements

andyaldersaggar maker's framemaker
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Hyperconverged is not a requirement of a vSAN, You can have a vSAN cluster that does nothing but provide storage and a separate compute cluster if you so desire. Paul may be more up to date on current offerings than me but as it's jskfan asking I suspect he wants to know what the term means.
Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
Andy, there are many semantics involved as you know and each vendor describes their solution how they want to.

For my 2 cents.

Hyperconverged (HCI) Virtual SAN: Compute and storage together.
Per Wikipedia:   Hyper-converged infrastructure ( HCI) is a  software-defined IT infrastructure that virtualizes all of the elements of conventional " hardware-defined" systems. HCI includes, at a minimum,  virtualized computing (a  hypervisor),  software-defined storage and virtualized networking ( software-defined networking). HCI typically runs on  commercial off-the-shelf (COTS) servers.

Virtual SAN: Software based SAN in many variations.  Storage is separated from compute hardware.  This would include Nextenta and others

Hardware SAN: Your standard SAN solution with physical controllers and LUNs on top

There are many grey areas since you have a lot of hardware based SANs porting their software to a software based SAN solution and technologies such as FoE allow you to use legacy protocols for new equipment.

Scalability wise the HCI solutions are constrained more than the two other solutions because you don't need to add hypervisors/compute to add more storage.  A good example of this is EMC VXRail where you have to add nodes, VMware licenses, etc.. to add more storage.  You then proceed to lose 50% or more with your storage due to 2X storage requirement and the requirement to keep a large portion of your vSAN unpopulated because you may lose storage capacity if you lose a node.  In a 5 node environment you have to keep 25% available while losing 50% right of the top.

There are use cases for every solution, the most important part is defining requirements and planning for your hardware cycle.

Author

Commented:
Paul,

The reason you chose vSAN is ,I believe, the performance, especially when the Drives are internal to the Hosts.
Now that you stated that Disks can be external, and connected to the Disk Adapter on the ESX Host.  I would like to know what kind of medium used to connect the external drive to the disk Adapter that makes it that fast.
To regular SAN usually ESX hosts connect via Ethernet,FC, FCoE,SCSCI  cables.
To vSAN what kind of cable used ? and what is the maximum length of it in order to keep high performance ?

Thanks
Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
The local drive aspect relating to performance doesn't necessarily apply since the VM's compute could be on host 1 and the storage of the vm could be on host 2 and then you are going over your network card just as you would with any other type of SAN.  Obviously the type of drives you are using make a difference as well network speed and switch configurations.  VMware vSAN uses a proprietary protocol, other vSAN vendors use everything from iSCSI, NFS, to connect but may use different protocols for replication if they have more than one node.  For instance FreeNAS or Openfiler can be considered vSANs and you can use iSCSI and NFS to connect to it.  
andyaldersaggar maker's framemaker
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
>Now that you stated that Disks can be external, and connected to the Disk Adapter on the ESX Host.

SAS cables about 3 foot long normally, performance is the same as if they were internal. There may be a tiny increase in latency due to the SAS expander but there are often expanders on internal backplanes too.
Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
With SAS-4 @ 22.5Gbit/s the latency is minimal.  Even 12Gb/s is more than most customer will ever need, The fanout cables for SAS allow you to add more shelves and/or provide redundancy.

Author

Commented:
3 foot-long? So the location of vSAN storage got to be very close to the location where ESX hosts are racked.


Author

Commented:
Paul,

vSAN : old meaning was about local disks on each ESX Host that can be shared between all ESX hosts. Now that Disks can be external instead of local, the old meaning does not apply anymore, since it is the same as having regular SAN, the disks are external to ESX Hosts somewhere in  separate racks, probably the only difference between  SAN  and vSAN is the performance because of the adapters used to connect ESX Hosts to Disks.
andyaldersaggar maker's framemaker
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
It's still local disks whether they are in internal cages or external ones directly attached to each host.

Look at this HBA for example - https://www.broadcom.com/products/storage/host-bus-adapters/sas-nvme-9400-8i8e
It has both internal and external connectors, the internal ones connect to internal disks and the external ones connect to external shelves but they behave just like the internal ones. That card is not supported by VMware vSAN but you could use it with other types of vSAN.

Author

Commented:
andyalder 

what makes vSAN different from regular SAN, if the disks can be external in the shelf just like regular SAN?

is the difference vSAN has a software that can make a bunch of disks act as one large disk?
andyaldersaggar maker's framemaker
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
a virtual SAN is a software SAN running in a virtual machine, a physical SAN is a hardware SAN. There's a bit of a grey area when it's a software SAN not running in a virtual machine, generally not though.

Author

Commented:
a virtual SAN is a software SAN running in a virtual machine, a physical SAN is a hardware SAN

you mean running in ESX host?
and performance is the same ? vSAN Cost is cheaper ?
saggar maker's framemaker
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
Yes it runs as a virtual machine on the host, that's how it can use local disks.

Performance can be better than a physical SAN as local disks are faster than remote ones.

As far as price goes VMware vSAN costs $2495 per CPU but StarWind vSAN is free up to 3 nodes. Hardware costs go up and down - no expensive dual-controller SAN but you need twice as many disks to mirror one vSAN to another for redundancy.

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.