Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Freenas replication is unusually slow.

Posted on 2014-10-28
3
Medium Priority
?
640 Views
Last Modified: 2014-11-18
Hi there,

I have 2 freenas systems configured as follows:

Build:      FreeNAS-9.2.1.8-RELEASE-x64 (e625626)
Platform      :   Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
Memory:      49122MB

PUSH - 1x28T raidz pool (2xSSD x 480GB for zilog, 1xSSD x 240GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
PULL - 1x28T raidz pool (2xSSD x 240GB for zilog, 1xSSD x 280GB + 13x3TB for data + 1x3TB for hotspare) Compression and deduplication are off.
1x22TB zvol is created on PUSH
Weekly snapshots.
zfs replication from PUSH to PULL is setup according freenas manual.
Doing initial replication which supposed to transfer about 22TBs of data.
filesystem ~85% full.

Machines are geographically separated with BW about 40 megabits/s over MPLS.

zfs replication takes too long to complete, it running only about 8 megabits/s (0.8-1 megabytes/s). The network is not the bottleneck.
Disabling cypher and compression for the ssh transport increased the speed to around 10 megabits/s (1-1.2 megabytes/s) which is obviously slow too.

Am I alone with this or is it a known issue? Does anyone even solved this kind of issues before?
Should I file a bug report?

Thanks!

Alex.
0
Comment
Question by:Nick Krapf
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 39

Accepted Solution

by:
Aaron Tomosky earned 2000 total points
ID: 40410840
Zfs send and receive has always been plagued by latency issues. Even on a lan I can significantly increase my snapshot sending speed by incorporating mbuffer into the mix.  This way the sending side doesn't have to wait for the receive side to write the data before sending more. I use a different distro than freenas, based on freebsd 9.2, so the scripts should work, but it wouldn't be part of the freenas GUI. Let me know if you are interested.
0
 

Author Comment

by:Nick Krapf
ID: 40435472
Yes, please share the script, possibly i will be able to rewrite it and integrate with freenas GUI.
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 40450449
So really the magic is in this one line:
zfs send -I $2@$OLDsnap $5 | ssh -c arcfour $4 "mbuffer -s 128k -m 1G | zfs receive -F $3"

if you install mbuffer on both the send and receive side, and use this command to do the sending, this goes MUCH faster.

Here is the full boat


make a folder /root/zfsrep and put the zfsrep3.sh script there
make a folder /root/zfsrep/zfsrep.snapshots

I call it with cron to a script that is something like this:
pools="tank/pool1 tank/pool2 tank2/pool1 tank/pool3"
remotehost="remote.domain.com"
for pool in $pools; do
	regex="^$pool@"
	currentsnapshot=`zfs list -H -o name -t snapshot | sort -r | grep $regex | head -1`
#patch for dest pool change	
if [ "$pool" = "tank2/pool1" ]; then
		destpool="tank1/tank2pool1"
	else
		destpool="$pool"
	fi
	echo "Syncing $pool on $currentsnapshot to $remotehost on $destpool"
	/root/zfsrep/zfsrep3.sh periodic $pool $destpool $remotehost $currentsnapshot
	#echo "done $pool"
done

Open in new window


I use a slightly hacked zfsnap to make my snapshots. The hack was to skip snapshots that had a hold on them when deleting
zfsrep3.sh
zfSnap
0

Featured Post

Migrating Your Company's PCs

To keep pace with competitors, businesses must keep employees productive, and that means providing them with the latest technology. This document provides the tips and tricks you need to help you migrate an outdated PC fleet to new desktops, laptops, and tablets.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Lets start to have a small explanation what is VAAI(vStorage API for Array Integration ) and what are the benefits using it. VAAI is an API framework in VMware that enable some Storage tasks. It first presented in ESXi 4.1, but only after 5.x sup…
Compliance and data security require steps be taken to prevent unauthorized users from copying data.  Here's one method to prevent data theft via USB drives (and writable optical media).
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
Suggested Courses

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question