Solved

Freenas replication is unusually slow.

Posted on 2014-10-28
3
430 Views
Last Modified: 2014-11-18
Hi there,

I have 2 freenas systems configured as follows:

Build:      FreeNAS-9.2.1.8-RELEASE-x64 (e625626)
Platform      :   Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
Memory:      49122MB

PUSH - 1x28T raidz pool (2xSSD x 480GB for zilog, 1xSSD x 240GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
PULL - 1x28T raidz pool (2xSSD x 240GB for zilog, 1xSSD x 280GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
1x22TB zvol is created on PUSH
Weekly snapshots.
zfs replication from PUSH to PULL is setup according freenas manual.
Doing initial replication which supposed to transfer about 22TBs of data.
filesystem ~85% full.

Machines are geographically separated with BW about 40 megabits/s over MPLS.

zfs replication takes too long to complete, it running only about 8 megabits/s (0.8-1 megabytes/s). The network is not the bottleneck.
Disabling cypher and compression for the ssh transport increased the speed to around 10 megabits/s (1-1.2 megabytes/s) which is obviously slow too.

Am I alone with this or is it a known issue? Does anyone even solved this kind of issues before?
Should I file a bug report?

Thanks!

Alex.
0
Comment
Question by:Nick Krapf
  • 2
3 Comments
 
LVL 38

Accepted Solution

by:
Aaron Tomosky earned 500 total points
ID: 40410840
Zfs send and receive has always been plagued by latency issues. Even on a lan I can significantly increase my snapshot sending speed by incorporating mbuffer into the mix.  This way the sending side doesn't have to wait for the receive side to write the data before sending more. I use a different distro than freenas, based on freebsd 9.2, so the scripts should work, but it wouldn't be part of the freenas GUI. Let me know if you are interested.
0
 

Author Comment

by:Nick Krapf
ID: 40435472
Yes, please share the script, possibly i will be able to rewrite it and integrate with freenas GUI.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
ID: 40450449
So really the magic is in this one line:
zfs send -I $2@$OLDsnap $5 | ssh -c arcfour $4 "mbuffer -s 128k -m 1G | zfs receive -F $3"

if you install mbuffer on both the send and receive side, and use this command to do the sending, this goes MUCH faster.

Here is the full boat


make a folder /root/zfsrep and put the zfsrep3.sh script there
make a folder /root/zfsrep/zfsrep.snapshots

I call it with cron to a script that is something like this:
pools="tank/pool1 tank/pool2 tank2/pool1 tank/pool3"
remotehost="remote.domain.com"
for pool in $pools; do
	regex="^$pool@"
	currentsnapshot=`zfs list -H -o name -t snapshot | sort -r | grep $regex | head -1`
#patch for dest pool change	
if [ "$pool" = "tank2/pool1" ]; then
		destpool="tank1/tank2pool1"
	else
		destpool="$pool"
	fi
	echo "Syncing $pool on $currentsnapshot to $remotehost on $destpool"
	/root/zfsrep/zfsrep3.sh periodic $pool $destpool $remotehost $currentsnapshot
	#echo "done $pool"
done

Open in new window


I use a slightly hacked zfsnap to make my snapshots. The hack was to skip snapshots that had a hold on them when deleting
zfsrep3.sh
zfSnap
0

Featured Post

Get up to 2TB FREE CLOUD per backup license!

An exclusive Black Friday offer just for Expert Exchange audience! Buy any of our top-rated backup solutions & get up to 2TB free cloud per system! Perform local & cloud backup in the same step, and restore instantly—anytime, anywhere. Grab this deal now before it disappears!

Join & Write a Comment

We wanted to provide an in-depth explanation of the Ping Node offering clarifications on its function and usage. Incorrect Ping Node configuration and functionality can cause problems with HA clusters. The importance of this article is critical for …
In this article, I provide some information on storage disks which go into calculations that will help you figure out how much Input/output Operations Per Second (IOPS) your disk subsystem can deliver. To effectively size & tune up applications l…
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now