Solved

Freenas replication is unusually slow.

Posted on 2014-10-28
3
503 Views
Last Modified: 2014-11-18
Hi there,

I have 2 freenas systems configured as follows:

Build:      FreeNAS-9.2.1.8-RELEASE-x64 (e625626)
Platform      :   Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
Memory:      49122MB

PUSH - 1x28T raidz pool (2xSSD x 480GB for zilog, 1xSSD x 240GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
PULL - 1x28T raidz pool (2xSSD x 240GB for zilog, 1xSSD x 280GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
1x22TB zvol is created on PUSH
Weekly snapshots.
zfs replication from PUSH to PULL is setup according freenas manual.
Doing initial replication which supposed to transfer about 22TBs of data.
filesystem ~85% full.

Machines are geographically separated with BW about 40 megabits/s over MPLS.

zfs replication takes too long to complete, it running only about 8 megabits/s (0.8-1 megabytes/s). The network is not the bottleneck.
Disabling cypher and compression for the ssh transport increased the speed to around 10 megabits/s (1-1.2 megabytes/s) which is obviously slow too.

Am I alone with this or is it a known issue? Does anyone even solved this kind of issues before?
Should I file a bug report?

Thanks!

Alex.
0
Comment
Question by:Nick Krapf
  • 2
3 Comments
 
LVL 39

Accepted Solution

by:
Aaron Tomosky earned 500 total points
ID: 40410840
Zfs send and receive has always been plagued by latency issues. Even on a lan I can significantly increase my snapshot sending speed by incorporating mbuffer into the mix.  This way the sending side doesn't have to wait for the receive side to write the data before sending more. I use a different distro than freenas, based on freebsd 9.2, so the scripts should work, but it wouldn't be part of the freenas GUI. Let me know if you are interested.
0
 

Author Comment

by:Nick Krapf
ID: 40435472
Yes, please share the script, possibly i will be able to rewrite it and integrate with freenas GUI.
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 40450449
So really the magic is in this one line:
zfs send -I $2@$OLDsnap $5 | ssh -c arcfour $4 "mbuffer -s 128k -m 1G | zfs receive -F $3"

if you install mbuffer on both the send and receive side, and use this command to do the sending, this goes MUCH faster.

Here is the full boat


make a folder /root/zfsrep and put the zfsrep3.sh script there
make a folder /root/zfsrep/zfsrep.snapshots

I call it with cron to a script that is something like this:
pools="tank/pool1 tank/pool2 tank2/pool1 tank/pool3"
remotehost="remote.domain.com"
for pool in $pools; do
	regex="^$pool@"
	currentsnapshot=`zfs list -H -o name -t snapshot | sort -r | grep $regex | head -1`
#patch for dest pool change	
if [ "$pool" = "tank2/pool1" ]; then
		destpool="tank1/tank2pool1"
	else
		destpool="$pool"
	fi
	echo "Syncing $pool on $currentsnapshot to $remotehost on $destpool"
	/root/zfsrep/zfsrep3.sh periodic $pool $destpool $remotehost $currentsnapshot
	#echo "done $pool"
done

Open in new window


I use a slightly hacked zfsnap to make my snapshots. The hack was to skip snapshots that had a hold on them when deleting
zfsrep3.sh
zfSnap
0

Featured Post

Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

AWS Glacier is Amazons cheapest storage option and is their answer to a ‘Cold’ storage service.  Customers primarily use this service for archival purposes and storage of infrastructure backups.  Its unlimited storage potential and low storage cost …
This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

740 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question