FreeNAS / NFS / VMWare tuning help
Posted on 2013-05-11
In a previous question I was asking for assistance in building a storage server for my VMWare ESXi host. Here's the end result:
- AMD FX6400 CPU / 16GB DDR3 RAM / FreeNAS 8.3.1 running off a USB stick
- 3x WD Red 2TB disks, running in a RAID1 mirror
- 2x ADATA 64GB SSDs, in a mirror, assigned as the ZFS log disks
- Intel 1000/GT gigabit network card, connected directly to ESXi host via crossover cable
- Onboard Realtek gigabit network port, connected to my switch and the rest of the network
VMWare ESXi Host:
- Supermicro 1U server / 2x Intel Xeon L5420 / 24GB DDR2 FB-DIMM / ESXi 5.1
- 1x 160GB hard drive for VMWare installation
- 2x onboard giganit network ports. 1x connected to the switch and the rest of the network. The other is connected directly to the Storage server via crossover cable.
Notes on the tuning / setup:
- The storage server has a couple ZFS datasets - one is for the VMs, the other two are used by the rest of the Windows network as a general fileserver
- All datasets are using LZJB compression, deduplication is turned off.
- VMWare connects to the Storage Server via NFS
- NFS Asynchronous mode is enabled
- The 3x datasets have a periodic snapshot running once every 6 hours which lasts 2 weeks. There are currently 64 snapshots total, it will rise up to 168 snapshots eventually.
I'm getting some reasonable performance:
- 65-70 MB/s read speeds according to HDTune installed on a Windows 7 VM
- 100-130 IOPS according to HDTune installed on a Windows 7 VM
When I SSH directly into the FreeNAS server and use DD to do some tests I get what IMHO are really excellent speeds, probably inflated due to the compression, RAM and SSDs:
// Writing a 20GB file of all zeros
dd if=/dev/zero of=test bs=2M count=10000
RESULT: 1434 MB/sec
// Reading a pre-created random file that is 8GB in size
dd if=/dev/random of=test bs=2M count=4000
dd if=test of=/dev/null bs=2M
RESULT: 524 MB/sec
// Making a copy of the pre-created random file
dd if=test of=test2 bs=2M
RESULT: 62 MB/sec <-- hmmmmm.....
I think the IOPS from VMWare is kind-of low but not I'm sure why. Also I think the throughput is decent, 65-70MB/s, but not fully saturating my gigabit network link, I'd expect closer to 90-100MB/s.
Does anyone have any suggestions for optimizing the equipment I've got and making the most out of it? Or, does anyone have any comments regarding the performance I'm getting - is it good? bad? great?