Solved

Nexenta on Commodity Hardware vs. NetApp/EMC

Posted on 2013-01-08
16
1,929 Views
Last Modified: 2016-11-23
Hi,

I am having a tough time deciding between getting the right storage for my company, we want it for a viruatlized VMware 5.x environment, and we want it to be rock solid.
I have extensive experience with NetApp and some EMC, and I wanted to get some feedback on how that would stack up against a system such as follows:

SuperMicro with 24Tb, LSI Logic cards controlling 6Tb of SSD and 18Tb of SATA 7.2K drives
Dual QLogic Fabric extenders and HBA's (8Gb Fiber)
Dell R610's for ESXi hosts
Nexenta zfs

vs.

NetApp clustered FAS3040 filers
2 Shelves of 4.2Tb each 15K 300Gb SAS Drives
Dual Broadcom 8Gb Fabric Switches, QLogic HBA's (8Gb Fiber)
Dell R610's for ESXi hosts

Thoughts?
0
Comment
Question by:sbsc
  • 5
  • 4
  • 3
  • +3
16 Comments
 
LVL 47

Accepted Solution

by:
dlethe earned 500 total points
Comment Utility
Apples and oranges.  I assume you are getting enterprise class storage, and ECC memory, so when it comes down to it, the 1s and 0s are on the same quality drives.

So reliability comes down to the level of redundancy you configure.  Either system can be configured for RAID0, or for a n-WAY RAID1 where all drives have exactly the same data being mirrored on multiple controllers, or something in the middle.

If you configure the FAS3040 so that each system isn't a mirror of the other, then you have a single point of failure just like you have with the Supermicro.

Bottom line, you can't go wrong with either ... as long as it is configured appropriately, you know how to maintain it (or pay somebody), and you have a good backup and disaster recovery plan.
0
 

Author Comment

by:sbsc
Comment Utility
Yes, almost the same quality drives, the supermicro has its main storage on SATA 7.2 drives while the NetAPP has all 300GB SAS Drives.  Also as said, the SuperMicro is a head+storage whereas the NetApp is controller-head and shelves.  if the supermicro board dies for example, the storage is down, whereas in the NetApp the other head will take over.  (Of course it will be in cluster mode, not two single filers)

I know I cant go wrong with either one, question is what are the pros/cons to each?
0
 
LVL 42

Expert Comment

by:paulsolov
Comment Utility
The main issue I have with commodity boxes is the level of monitoring.  Hopefully the supermicro comes with bmc card so that you can actually tell that a drive went down, otherwise you don't have insight in a good bit of your environment.  Had a customer with this type of solution with Openfiler that went down about a week ago and had to restore from tape.

I'm not sure about the Nextenta,is it active/active or active/passive?  If you have a lot of read/write data it may make sense to get slower drives and get a shelf with 6 to 8 SSD cards for flex pool configuration.  

ZFS is very similar to WAFL with respect to functionality but you may also want to look at SAN integration with your applications such as vmware, sql, oracle, exchange, etc.., wasn't sure if Nextenta offers the same type of options.

At the end of the day iops are iops, both solutions will do the job.
0
 

Author Comment

by:sbsc
Comment Utility
I'm not so familiar with Nexenta either... that's part of the problem! :) I am familiar with NetApp and I know the manageability and monitoring which is a strong point against the SuperMicro in my opinion.  I dont know if it comes with a BMC card, and for sure it doesnt have Autosupport....

Nexenta is supposed to be fully integrate able to VMware, and it gives a datastore to the ESXi, from what I understand it also can copy on a block level from one system to another sort of like snapmirror.
0
 
LVL 42

Expert Comment

by:paulsolov
Comment Utility
Looks like supermicro may come with a bmc card that will do snmp, smtp, etc.. what I don't know is what type of traps it provides and is it smart enough to monitor the raid controller (other than the built in one), if it can't then that would be a manual check on a daily basis and hopefully you have a spare to rebuild on.

http://www.supermicro.com/products/nfo/ipmi.cfm
0
 
LVL 47

Assisted Solution

by:dlethe
dlethe earned 500 total points
Comment Utility
Nexenta is basically Solaris with a GUI.  But i beg to differ with you on the controller-head-storage.  NetApp uses a unix-kernel and software RAID, just like Nexenta. The difference is that Nexenta is an open O/S.

You can use Nexenta to run applications on it, so you can get much more bang for the buck when you run apps on it like database.   You do NOT want to use ESXi on it.  You want the entire machine to run Solaris.  ESXi is not efficient when it comes to disk I/O.

Do NOT buy a built in RAID controller. ZFS + Solaris is your RAID controller.  YOu really need to research ZFS. It has much more functionality like hot snapshot, iSCSI targets, online expansion, resilvering to expand, load balancing ... you can even do clustering if you want (but that is an upgrade with Nexenta).

Plus supermicro makes quality servers. I know of an ISP that has over 100,000 of them. You're familiar with the company too, but I just can't tell you due to NDA.  Another customer of mine is a cloud provider in Asia. They run nexenta on their systems. The computers are not supermicro, but they are all running nexenta.  Each system has 200-250 Seagate SAS disks on them.   (with 4-8 LSI controllers).

Make sure you buy several LSI controllers, and be sure to dual port the enclosures to get load balancing and path failover also.
0
 

Author Comment

by:sbsc
Comment Utility
what I meant is that the architecture of NetApp is to have a head unit, like say an FAS2020 and then have shelves.  Same goes for EMC, and many others.  you can then have 2 heads in a cluster active/passive, or with the newer OnTap 8.1 cluster-mode with namespaces etc.  

Databases such as Oracle RAC use a "raw" file system on a storage, in that case it may be better to use a bare metal server with an HBA to the fabric.  btw, what do you mean ESXi is not efficient when it come to disk IO?
0
 
LVL 47

Assisted Solution

by:dlethe
dlethe earned 500 total points
Comment Utility
Yes, ESXi is not efficient, or more correctly, can't be as efficient at disk I/O when it virtualizes the disk. By definition, when I/O is combined to improve throughput then latency suffers, and vice-versa.  A native O/S application's know more about the next I/O requests than ESXi can ever know.

ZFS does a flush-on-write.  This is how it gets around the infamous RAID5 write hole.  Outside of ZFS file systems, when a record / block is updated, it is rewritten, and hopefully it is cached.

But in zfs, data is always written in a NEW place, that is guaranteed not to be cached.  Then once the write completes, it flushes, to insure it goes to disk. Then after zfs knows it has been written, it marks the old data as free space.    So, you never get any cached writes, and all writes are flushed.    (ZFS gets the speed by integrating SSDs into a ZIL and read cache.  Also ZFS uses system RAM, (as much as it can get) for read cache.  

ZFS also does variable sized I/O.   If it needs 15 blocks, then that is what it asks for.  But virtualized systems won't read just 15 blocks when a VM asks for it.  It will try to second-guess and get 16 blocks (well, more likely 64) by doing that, it not only adds latency, but reads go into the bitbucket and fill up cache that is better used elsewhere.  So that is why you turn off HDD prefetching on ZFS systems.

These are just a few reasons why one does not virtualize a ZFS system. (BUt reasonable and low overhead to virtualize the root pool. Just nail the controllers and physical drives for all the storage via vmdirect io, so no virtualized I/O gets done to the disk drives.)
0
Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

 
LVL 9

Expert Comment

by:FilipZahradnik
Comment Utility
Detailed technical arguments aside, dedicated storage vendors can offer a higher level of support across their hardware and software together. This is because they control and supply both the hardware and the software. It might be hard to match the service level by combining software support from one vendor with hardware support from another.

So if the criterion is 'rock solid storage', I would go with a dedicated storage vendor.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
with all the after market support and integration between NetApp and VMware, I would vote NetApp if you want a "rock solid solution"
0
 
LVL 28

Expert Comment

by:jhyiesla
Comment Utility
Not to muddy up the issue, but you may also want to look at Nimble Storage. We've looked at both Nexenta and Nimble and found them both, at least technology-wise, to be "similar" in their approach of caching and SSD and algorithms to help make the IOPs better.  At least from the quotes that we have received, Nimble was quite a bit less expensive.
0
 

Author Comment

by:sbsc
Comment Utility
It looks like the general consensus is to go with NetApp vs. the Nexenta.  dlethe or anyone else, do you think the performance on the SuperMicro/Nexenta could blow away any of the
systems by NetAPP/EMC? that's at least what I was told, that they had put this system together for some high traffic websites to get some big throughput and performance. Granted this isn't as secure/enterprise but with enough hardware/software question is if it can reach a similiar stability and if the performance really blows away these vendors like NetApp/EMC.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
ZFS will possibly outperform, but it does not have the Integration, Support and Software thay NetApp/EMC currently has.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
Nexenta has a channel too.  Not only that, but let's face it, the author isn't going to make a dent in the quota of an EMC/NetApp rep, so they'll be dealing with a reseller channel, not EMC/NetApp directly.

Those systems are also closed and for the price delta, one can add a lot of extra redundancy and make the ZFS solution bullet-proof.  BUT I must point out that one just can't throw any system together.  The right make/model of SSDs; HDDs; HDD firmware; enclosure/backplane; controllers; and topology matter.   That is why EMC/NetApp make the big bucks.  They qualify and test what works together.

The people I deal with you have had unstable  Nexenta systems didn't buy the right hardware and didn't configure it properly.  They viewed all "enterprise class" HDDs as the same, and felt same way about SSDs & controllers.
0
 
LVL 42

Expert Comment

by:paulsolov
Comment Utility
Actually there's a decent price war going on between EMC and Netapp, Seems like both are in a refresh cycle and there are some deep discounts and trade in deals.  I think both are looking more towards market share since they make more on support and licenses than on hardware for a 3 year cycle.  Always play the vendors against each other, can't lose no matter who you go with.

I would see if you can get a system to kick the tires on, I have seen EMC and Netapp do this, I wonder if Nextenta would do the same.  A lot of vendors are also supporting openstack which is interesting storage solutions, seems to be gaining traction
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
Nexenta is an operating system, and one can download the open source version and try it for free.  You can run it in a VM or on a windows laptop if you want to play with it. Nexenta doesn't sell hardware.  

 Equate this as you are evaluating that supermicro gear running Win2K8. Would you call Microsoft and ask them to send you that supermicro system with the disks?   No, but you can download Win2K8 trials (or Win2K12 trials) and run it on your own gear to kick the tires.  Nexenta has that too.
0

Featured Post

Free book by J.Peter Bruzzese, Microsoft MVP

Are you using Office 365? Trying to set up email signatures but you’re struggling with transport rules and connectors? Let renowned Microsoft MVP J.Peter Bruzzese show you how in this exclusive e-book on Office 365 email signatures. Better yet, it’s free!

Join & Write a Comment

This is an issue that we can get adding / removing permissions in the vCSA 6.0. We can also have issues searching for users / groups in the AD (using your identify sources). This is how one of the ways to handle this issues and fix it.
Veeam Backup & Replication has added a new integration – Veeam Backup for Microsoft Office 365.  In this blog, we will discuss how you can benefit from Office 365 email backup with the Veeam’s new product and try to shed some light on the needs and …
Teach the user how to use vSphere Update Manager to update the VMware Tools and virtual machine hardware version Open vSphere Client: Review manual processes for updating VMware Tools and virtual hardware versions: Create a new baseline group in vSp…
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now