Solved

Freenas replication is unusually slow.

Posted on 2014-10-28
3
587 Views
Last Modified: 2014-11-18
Hi there,

I have 2 freenas systems configured as follows:

Build:      FreeNAS-9.2.1.8-RELEASE-x64 (e625626)
Platform      :   Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
Memory:      49122MB

PUSH - 1x28T raidz pool (2xSSD x 480GB for zilog, 1xSSD x 240GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
PULL - 1x28T raidz pool (2xSSD x 240GB for zilog, 1xSSD x 280GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
1x22TB zvol is created on PUSH
Weekly snapshots.
zfs replication from PUSH to PULL is setup according freenas manual.
Doing initial replication which supposed to transfer about 22TBs of data.
filesystem ~85% full.

Machines are geographically separated with BW about 40 megabits/s over MPLS.

zfs replication takes too long to complete, it running only about 8 megabits/s (0.8-1 megabytes/s). The network is not the bottleneck.
Disabling cypher and compression for the ssh transport increased the speed to around 10 megabits/s (1-1.2 megabytes/s) which is obviously slow too.

Am I alone with this or is it a known issue? Does anyone even solved this kind of issues before?
Should I file a bug report?

Thanks!

Alex.
0
Comment
Question by:Nick Krapf
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 39

Accepted Solution

by:
Aaron Tomosky earned 500 total points
ID: 40410840
Zfs send and receive has always been plagued by latency issues. Even on a lan I can significantly increase my snapshot sending speed by incorporating mbuffer into the mix.  This way the sending side doesn't have to wait for the receive side to write the data before sending more. I use a different distro than freenas, based on freebsd 9.2, so the scripts should work, but it wouldn't be part of the freenas GUI. Let me know if you are interested.
0
 

Author Comment

by:Nick Krapf
ID: 40435472
Yes, please share the script, possibly i will be able to rewrite it and integrate with freenas GUI.
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 40450449
So really the magic is in this one line:
zfs send -I $2@$OLDsnap $5 | ssh -c arcfour $4 "mbuffer -s 128k -m 1G | zfs receive -F $3"

if you install mbuffer on both the send and receive side, and use this command to do the sending, this goes MUCH faster.

Here is the full boat


make a folder /root/zfsrep and put the zfsrep3.sh script there
make a folder /root/zfsrep/zfsrep.snapshots

I call it with cron to a script that is something like this:
pools="tank/pool1 tank/pool2 tank2/pool1 tank/pool3"
remotehost="remote.domain.com"
for pool in $pools; do
	regex="^$pool@"
	currentsnapshot=`zfs list -H -o name -t snapshot | sort -r | grep $regex | head -1`
#patch for dest pool change	
if [ "$pool" = "tank2/pool1" ]; then
		destpool="tank1/tank2pool1"
	else
		destpool="$pool"
	fi
	echo "Syncing $pool on $currentsnapshot to $remotehost on $destpool"
	/root/zfsrep/zfsrep3.sh periodic $pool $destpool $remotehost $currentsnapshot
	#echo "done $pool"
done

Open in new window


I use a slightly hacked zfsnap to make my snapshots. The hack was to skip snapshots that had a hold on them when deleting
zfsrep3.sh
zfSnap
0

Featured Post

Why Off-Site Backups Are The Only Way To Go

You are probably backing up your data—but how and where? Ransomware is on the rise and there are variants that specifically target backups. Read on to discover why off-site is the way to go.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article we will learn how to backup a VMware farm using Nakivo Backup & Replication. In this tutorial we will install the software on a Windows 2012 R2 Server.
A look at what happened in the Verizon cloud breach.
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
Suggested Courses

628 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question