Question

RSYNC and compression issue - should I use it or no? Then how?

Asked by: akohan



Hello,

I have written a backup/recovery script and still working on it but still there is something, which I'm not satisfied with it yet. The whole theory of RSYNC is synchronization or data transfer so why do some or maybe most of the Admins mention it as backup tool?

Some believe tar or dump or any compression must not be used during rsync since it causes some confusion in its process and some tar or compress their files before calling rsync.

If I don’t use tar or compression then I'm having problem with storage (room)
If I use it then I defeat its theory and its original definition.

1 ) What do you suggest?
2) Is there any way to have RSYNC to make the target directory on target media? at  this time I use mkdir to make it before calling rsync.
3) Is it possible target a DVD or CD as media using rsync?
4) How to show a progress bar during its tranfer?


Your help is appreciated greatly.

AKOHAN

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2006-09-03 at 03:12:10ID21976677
Tags

rsync

,

compression

Topic

Linux Administration

Participating Experts
4
Points
250
Comments
11

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. Compression!!!
    I need to write a file compression routine in VB6 (!). Speed is not an issue, file security and space saving are however. I realise this a large and vague area and that it may be 'out of place' on such a site. I am hoping that perhaps source code is already available on which...
  2. compression
    I want to write my own compression program. I want just be able to compress a byte array. (I load a file into a byte array and send it to my sub and it compresses it.) Basicly just like zlib does! I someone could just give me a few pointers... Thanks
  3. Difference between gzip, compress (tar)
    What is the difference between gzip and compress(tar) in Unix operating system?
  4. tar with hardware compression
    Hello, I wonder if anyone can help me with a tape issue. I'm using tar to write files to a DLT4 tape, capacity 40/80. The drive has hardware compression enabled. Does anyone know of a way to see how well an archive has compressed and thus how much of the tape it actually ...
  5. Tape Drive Travan TR7: is it better to write to the tape a t…
    I've just bought a Certance Travan TR7 20/40GB (uncompressed/compressed). I want to know which one is the best and cleanest apporach to do a backup under LINUX (Debian Linux). Actually I just send a tar to the tape, using this command: tar cvf /dev/ht0 directorytobackup/ If...

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: slow1000Posted on 2006-09-03 at 14:20:22ID: 17446593

Although this really doesn't answer your question directly, I think Backup PC or another utility designed for backups might be easier for you to use, and you'll be happy with the results.
http://backuppc.sourceforge.net/info.html
http://www.bacula.org/
http://www.amanda.org/

 

by: akohanPosted on 2006-09-04 at 11:22:58ID: 17451136


Hello Slow1000,

Thank you for your comment. Yes, they are good backup/recovery softwares but I also need to use RSYNC for a synchronization operation. Once again thanks.

AKOHAN

 

by: bryanlloydharrisPosted on 2006-09-05 at 10:10:43ID: 17457323

Is it possible to use a compressed file system?  This would compress the partition but would not appear to be compressed for operations such as rsync.  This might be a good choice for compressing the data, if it is transparent to rsync.

 

by: akohanPosted on 2006-09-05 at 10:24:00ID: 17457409


Hello,

Would you please explain it more clear? what method? what tool are you talking about?

Thanks,
AKOHAN

 

by: pjedmondPosted on 2006-09-06 at 05:53:36ID: 17462705

2 approaches are available to you:

1.   Tar up the files before transferring the files concerned - use bz2 or gz compression prior to rsynching between the systems. Disadvantage of this is that you have to transfer all the files in the bz2/gz archive if only one file in the archive has changed.

2.    Compress the rsynch stream:

rsync -Cavz -e ssh user@source.ip user@dest.ip

Obviously you may not need to use ssh as the shell for the transfer, and also source and/or dest may be local, so you wont need the long form.

man rsynch

to find required syntax for your circumstances.

(   (()
(`-' _\
 ''  ''

 

by: pjedmondPosted on 2006-09-06 at 05:56:26ID: 17462724

3) Is it possible target a DVD or CD as media using rsync?

You can *IF* the DVD or CD is configured as a RAM disc.

4) How to show a progress bar during its tranfer?

With difficulty...and it won't be accurate anyway! Imaging 1000 files, and all are 1 byte in size, except the 1000th on which is 5GB...the progress will only show progress through number of files, and will therefore almost immediately after starting be at 99.9% complete, and remain that way until completion.

(   (()
(`-' _\
 ''  ''

 

by: akohanPosted on 2006-09-14 at 15:23:25ID: 17524921


Hello,

How couldI configure a CD or DVD as RAM disc?

Any clue?

Thanks,
AKOHAN

 

by: Cyclops3590Posted on 2006-12-03 at 11:04:34ID: 18064060

Moderator,

I want to also make clear that I'd recommend a grade of C at best since the solution only led the asker in the right direction and didn't provide a full, to-the-end, solution.

Cyclops3590
EE Cleanup Volunteer

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...