Solved

FreeNAS / NFS / VMWare tuning help

Posted on 2013-05-11
6
2,002 Views
Last Modified: 2013-05-18
Hi guys,

In a previous question I was asking for assistance in building a storage server for my VMWare ESXi host. Here's the end result:

FreeNAS Server:
      - AMD FX6400 CPU / 16GB DDR3 RAM / FreeNAS 8.3.1 running off a USB stick
      - 3x WD Red 2TB disks, running in a RAID1 mirror
      - 2x ADATA 64GB SSDs, in a mirror, assigned as the ZFS log disks
      - Intel 1000/GT gigabit network card, connected directly to ESXi host via crossover cable
      - Onboard Realtek gigabit network port, connected to my switch and the rest of the network

VMWare ESXi Host:
      - Supermicro 1U server / 2x Intel Xeon L5420 / 24GB DDR2 FB-DIMM / ESXi 5.1
      - 1x 160GB hard drive for VMWare installation
      - 2x onboard giganit network ports. 1x connected to the switch and the rest of the network. The other is connected directly to the Storage server via crossover cable.

Notes on the tuning / setup:

    - The storage server has a couple ZFS datasets - one is for the VMs, the other two are used by the rest of the Windows network as a general fileserver
    - All datasets are using LZJB compression, deduplication is turned off.
    - VMWare connects to the Storage Server via NFS
    - NFS Asynchronous mode is enabled
    - The 3x datasets have a periodic snapshot running once every 6 hours which lasts 2 weeks. There are currently 64 snapshots total, it will rise up to 168 snapshots eventually.

I'm getting some reasonable performance:

    - 65-70 MB/s read speeds according to HDTune installed on a Windows 7 VM
    - 100-130 IOPS according to HDTune installed on a Windows 7 VM

When I SSH directly into the FreeNAS server and use DD to do some tests I get what IMHO are really excellent speeds, probably inflated due to the compression, RAM and SSDs:

// Writing a 20GB file of all zeros
   dd if=/dev/zero of=test bs=2M count=10000
RESULT:     1434 MB/sec
   
// Reading a pre-created random file that is 8GB in size
   dd if=/dev/random of=test bs=2M count=4000
   dd if=test of=/dev/null bs=2M
RESULT:    524 MB/sec

// Making a copy of the pre-created random file
   dd if=test of=test2 bs=2M
RESULT:    62 MB/sec   <-- hmmmmm.....


I think the IOPS from VMWare is kind-of low but not I'm sure why. Also I think the throughput is decent, 65-70MB/s, but not fully saturating my gigabit network link, I'd expect closer to 90-100MB/s.

Does anyone have any suggestions for optimizing the equipment I've got and making the most out of it? Or, does anyone have any comments regarding the performance I'm getting - is it good? bad? great?
0
Comment
Question by:Frosty555
  • 4
6 Comments
 
LVL 28

Accepted Solution

by:
asavener earned 500 total points
ID: 39159003
You might gain a little performance by enabling jumbo frames.

Also, VMware's recommendation is that snapshots not be kept for longer than 72 hours.  You can run into some serious performance problems (and increased disk usage) if you keep snapshots.
0
 
LVL 28

Expert Comment

by:asavener
ID: 39159007
If your network is not saturated, then your performance is probably limited by the underlying hardware of your FreeNAS system.
0
 
LVL 28

Assisted Solution

by:asavener
asavener earned 500 total points
ID: 39159008
VMWare ESXi Host:
      - Supermicro 1U server / 2x Intel Xeon L5420 / 24GB DDR2 FB-DIMM / ESXi 5.1
      - 1x 160GB hard drive for VMWare installation
      - 2x onboard giganit network ports. 1x connected to the switch and the rest of the network. The other is connected directly to the Storage server via crossover cable.
You have a serious lack of redundancy for a production environment.  Especially running ESXi off of a single disk.
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 28

Expert Comment

by:asavener
ID: 39159010
I misread about snapshots; I thought you were talking about VMware snapshots, not filesystem snapshots.
0
 
LVL 120
ID: 39159298
More disks in the ZFS pool, SSD for ZIL and ARC caches, try iSCSI it might be faster!

Oracle would have been a better solution! ZFS performs better and has more features.
0
 
LVL 31

Author Closing Comment

by:Frosty555
ID: 39177525
The problem was ZFS snapshots (note: NOT VMWare snapshots, I didn't have any of those, but I had a periodic snapshot task running for ZFS) - I had over 60 of them and after I deleted them, my IOPS jumped up to 4000 and I was getting 100MB/s read speeds.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Separating Management Network 6 82
Cloning Solid State to Hybrid 7 36
Find ESXI host from within guest 16 82
VMWare vSAN 5 22
In this article, I will show you HOW TO: Suppress Configuration Issues and Warnings Alert displayed in Summary status for ESXi 6.5 after enabling SSH or ESXi Shell.
This article aims to explain the working of CircularLogArchiver. This tool was designed to solve the buildup of log file in cases where systems do not support circular logging or where circular logging is not enabled
Advanced tutorial on how to run the esxtop command to capture a batch file in csv format in order to export the file and use it for performance analysis. He demonstrates how to download the file using a vSphere web client (or vSphere client) and exp…
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question