Question

rsync under cygwin too slow

Asked by: vinod

I am using rsync under bash/cygwin on my laptop to backup Documents folder from laptop at home to my desktop in my office, both running Vista+SP1. From laptop, I VPN to my office over broadband, map shared folder at desktop as network drive Z on my laptop. The command used is:

rsync -rtuplg --stats --verbose --progress --modify-window=2 /cygdrive/c/Users/vinod/Documents /cygdrive/z/backups/

The backups are working but process is much too slow. As you may see from the statistics given below, the transfer rate was more than 100 times slower than the available bandwidth. I had only one new file on the laptop, only 43 KB in size. The folder has about 700 other files totalling about 120 MB in size.  The statistics shows it took only 0.358 seconds to generate+transfer file list, but actually it took more than 300 seconds. I know there are overheads due to VPN, File sharing, cygwin etc. which could account for a factor 3x or so, but 100x no way. What am I doing wrong?

Post rsync statistics:

Number of files: 723
Number of files transferred: 1
Total file size: 120133912 bytes
Total transferred file size: 43008 bytes
Literal data: 43008 bytes
Matched data: 0 byes
File list size: 19913
File list generation time: 0.358 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 67295
Total bytes received: 4350

sent 67295 bytes  received 4350 bytes 232.24 bytes/sec
total size is 120133912  speedup is 1676.79

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2009-03-16 at 21:27:09ID24236450
Tags

rsync cygwin

Topics

rSync Backup Utility

,

CYGWIN

Participating Experts
4
Points
500
Comments
12

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. Rsync not logging - using cygwin over ssh
    This is driving me crazy, have googled for hours! I have a win2k3 server running cygwin. I am using rsync to backup files to a linux server at another site. Here is the command I use in Cygwin (with some mods to protect the innocent) rsync -av --no-whole-file --delete --e...
  2. Cygwin over VPN
    Hello Everybody I just installed Cygwin on my PC (Windows XP SP2) and I have to access a Unix server (Tru64) over a VPN Connection. I'm using a checkpoint vpn client. My vpn account have all the privileges on the remote network. I can access the remote server using TELNET an...
  3. using CYGWIN
    I know this is a reach. But I am desperate. I am trying to establish and xsession in order to gain a terminal connection to our client servers. I installed cygwin. I am currently able to remotely log in using putty to our client servers by logging into our ftp server and then...
  4. Vista and Cygwin
    Will cygwin run on Vista? any Issues or concerns doing so?
  5. CYGWIN/RSYNC - Email Notification of files transfered
    I've got rsync sync files between multiple servers nightly using cron tab. Is there a way to send out an email notification of files transfered and/or if it was a sucess or failure? This is using cygwin on a windows 2003 server.

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: vinodPosted on 2009-03-17 at 21:12:38ID: 23915476

BTW, I noticed from the network counters that in order to backup this 43 KB file, rsync consumed a whopping 100 MB of downloaded and 12 MB of uploaded bytes. I can't say rsync is saving network bandwidth when it is consuming more than 2000 times the payload. Any expert opinion?

Any taker at 500 points?

Vinod

 

by: woolmilkporcPosted on 2009-03-18 at 06:14:34ID: 23918258

Hi,

this could be related to MTU size. If the MTU size your laptop uses is bigger than the size supported by your broadband connection, it comes to fragmentation. Use ping with fragmentation disabled (-f) and a given packet size (-l) -

ping [office] -f -l 1492

If the message 'Packet needs to be fragmented but DF set.' appears, decrease the packet size until the message doesn't show up anymore. You should use the size determined that way.

You can change the MTU settings in Vista using the following command from the command line as an admin -

netsh interface ipv4 set subinterface "Local Area Connection" mtu=[new_mtu] store=persistent

If you just want to see what your settings are you can run.

netsh interface ipv4 show subinterfaces

HTH

wmp


 

 

by: woolmilkporcPosted on 2009-03-18 at 07:28:25ID: 23919062

It might also be related to Vista's auto-tuning feature. See this article -

http://support.microsoft.com/kb/934430

 

by: vinodPosted on 2009-03-18 at 13:25:57ID: 23923314

Woolmilkporc,

Nice thought but I did not help. I don't think mtu or tcp autotuninglevel is causing the problem. I tried normal cp of files, they work just fine and the transfer rate is as per expectations. The problem is shear volume of matadata rsync requires to get info about file time stamps and size etc. It transfered 110 MB of matadata when the whole directory tree had only 100 MB of all files. Transferring 1 MB of matadata to transfer 10 MB of file data and dropping the rest of drop 90 MB makes sense. But transferring 100 MB of matadata to transfer 1 MB of file data is insane.

I did ls -lR on /cygdrive/z/backups/, sure enough it transferred 100 MB over the network while the payload (listing text) was only 60 KB. I experimented two linux NFS client/server machines. I did ls -lR on a directory with very similar contents and network counters show only 0.5 MB were transferred. It must be some weird combination of rsync, cygwin and Windows that is causing problem.

Vinod

 

by: ai_ja_naiPosted on 2009-03-23 at 14:24:10ID: 23962468

>BTW, I noticed from the network counters that in order to backup this 43 KB file, rsync consumed a whopping 100 MB of downloaded and
>12 MB of uploaded bytes. I can't say rsync is saving network bandwidth when it is consuming more than 2000 times the payload. Any
>expert opinion?

yes, that you are not transferring just 43k, but 100MB... review the folders for hidden files

 

by: vinodPosted on 2009-04-19 at 08:29:12ID: 24179342

There are no hidden folders.

 

by: ai_ja_naiPosted on 2009-04-19 at 08:48:03ID: 24179403

My opinion is that rsync did not generate those 100/12 MB of traffic. The report clearly states that 67kB were sent, on an overall of 120MB of data.
BTW, instead of -rtuplg use -a (which is the same, minus -u, which you don't need).

 

by: vinodPosted on 2009-04-19 at 15:06:56ID: 24180779

100MB of data was indeed transfered as a result of rsync. I monitored the network counters repeatedly. They start moving in sync with rsync.

 

by: ai_ja_naiPosted on 2009-04-20 at 02:11:13ID: 24182899

Ok, would you post rsync's stat logs when you see it's tranferring 100MB? As you did in the opening post
Oh, please, post the network counters also

 

by: bdb112Posted on 2009-04-21 at 22:31:38ID: 24201601

Seems like you are using rsync in the wrong way - rsync is meant to be a thin pipe between two intelligent nodes which bothe have cheap access to their files.  In your case, both the local and the remote nodes are your computer, and the "remote" node is accessing its data over a network, which is totally against the principle, because the "remote" node is examining its whole file system over the net.

Two solutions:
(i) Run rsync the way it was intended, with a daemon on the remote node (i.e. Don't use the network mount Z as the target).  This requires rsync to run on the target - i.e. you need cygwin there or a linux box.
If you have admin privileges there, I can give you more info.

(ii) The lazy/easy way for small files - but inefficient - run rsync based on modification times.  This will be much slower than (i), is a little less foolproof, but will be way better than what you are doing.
The problem here is that this seems to be what you are doing (you would use -c to force a checksum comparison, which would be foolish on a network drive).  checks on file dates and size should only need 723* ypically a 4k byte header at the most - 3MB, more likely 723*256bytes.  Which makes me think your remote file mount is not very clever.

If you want to pursue this lesser alternative, you need to say how the file is mounted - (i) is by far preferable)

 

by: bharatpurPosted on 2009-05-14 at 03:30:46ID: 24383303

Hey you can use rsync directly by installing their executable available for windows.
Download "cwRsync_3.1.0_Installer.zip" and "cwRsyncServer_3.1.0_Installer.zip" from below link:
http://sourceforge.net/project/showfiles.php?group_id=69227&package_id=68081

For more info on these setups check :
http://www.itefix.no/i2/cwrsync

and for vista compatibility (if u face some error while installing) check: http://www.itefix.no/i2/node/11623

Install cwRsyncServer_3.1.0_Installer.zip at ur office desktop PC and cwRsync_3.1.0_Installer.zip on your laptop and play with them....

You need to map the network drive on your office PC and try using the same rsync command given by you and u can try with less ovehead by reducing options for it..

 

by: ai_ja_naiPosted on 2009-05-14 at 03:42:48ID: 24383371

bharatpur, read the question

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...