Solved

Freenas replication is unusually slow.

Posted on 2014-10-28
3
526 Views
Last Modified: 2014-11-18
Hi there,

I have 2 freenas systems configured as follows:

Build:      FreeNAS-9.2.1.8-RELEASE-x64 (e625626)
Platform      :   Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
Memory:      49122MB

PUSH - 1x28T raidz pool (2xSSD x 480GB for zilog, 1xSSD x 240GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
PULL - 1x28T raidz pool (2xSSD x 240GB for zilog, 1xSSD x 280GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
1x22TB zvol is created on PUSH
Weekly snapshots.
zfs replication from PUSH to PULL is setup according freenas manual.
Doing initial replication which supposed to transfer about 22TBs of data.
filesystem ~85% full.

Machines are geographically separated with BW about 40 megabits/s over MPLS.

zfs replication takes too long to complete, it running only about 8 megabits/s (0.8-1 megabytes/s). The network is not the bottleneck.
Disabling cypher and compression for the ssh transport increased the speed to around 10 megabits/s (1-1.2 megabytes/s) which is obviously slow too.

Am I alone with this or is it a known issue? Does anyone even solved this kind of issues before?
Should I file a bug report?

Thanks!

Alex.
0
Comment
Question by:Nick Krapf
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 39

Accepted Solution

by:
Aaron Tomosky earned 500 total points
ID: 40410840
Zfs send and receive has always been plagued by latency issues. Even on a lan I can significantly increase my snapshot sending speed by incorporating mbuffer into the mix.  This way the sending side doesn't have to wait for the receive side to write the data before sending more. I use a different distro than freenas, based on freebsd 9.2, so the scripts should work, but it wouldn't be part of the freenas GUI. Let me know if you are interested.
0
 

Author Comment

by:Nick Krapf
ID: 40435472
Yes, please share the script, possibly i will be able to rewrite it and integrate with freenas GUI.
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 40450449
So really the magic is in this one line:
zfs send -I $2@$OLDsnap $5 | ssh -c arcfour $4 "mbuffer -s 128k -m 1G | zfs receive -F $3"

if you install mbuffer on both the send and receive side, and use this command to do the sending, this goes MUCH faster.

Here is the full boat


make a folder /root/zfsrep and put the zfsrep3.sh script there
make a folder /root/zfsrep/zfsrep.snapshots

I call it with cron to a script that is something like this:
pools="tank/pool1 tank/pool2 tank2/pool1 tank/pool3"
remotehost="remote.domain.com"
for pool in $pools; do
	regex="^$pool@"
	currentsnapshot=`zfs list -H -o name -t snapshot | sort -r | grep $regex | head -1`
#patch for dest pool change	
if [ "$pool" = "tank2/pool1" ]; then
		destpool="tank1/tank2pool1"
	else
		destpool="$pool"
	fi
	echo "Syncing $pool on $currentsnapshot to $remotehost on $destpool"
	/root/zfsrep/zfsrep3.sh periodic $pool $destpool $remotehost $currentsnapshot
	#echo "done $pool"
done

Open in new window


I use a slightly hacked zfsnap to make my snapshots. The hack was to skip snapshots that had a hold on them when deleting
zfsrep3.sh
zfSnap
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article aims to explain the working of CircularLogArchiver. This tool was designed to solve the buildup of log file in cases where systems do not support circular logging or where circular logging is not enabled
Many businesses neglect disaster recovery and treat it as an after-thought. I can tell you first hand that data will be lost, hard drives die, servers will be hacked, and careless (or malicious) employees can ruin your data.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

751 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question