Advertisement

03.31.2008 at 09:08AM PDT, ID: 23282990
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

Compressed backup (compress or pkzip) with tar and/or cpio
Tags: IBM, AIX, 5.3, Shell Unix KSH
Hi!

I need a help to create two(2) shell script to speed up backup and restore and optimize the ammount
used on tape. I use TAPE IBM LTO Ultrium 3 (using hardware compression) in a server RISC IBM p520 4 proc of 1,5 ghz, 2 GB mem, 4 disks of 143 GB 15Krpm.  The backup of 33 GB has lasted about 1 hour and I think I can speed up the process improving the software compression. System is idle on moment
of backup.


Shell Script 1- compress and backup a full directory recursivelly (all files) or individual file directly to the tape (like tar).
One line script with pipe(s) would be great (to improve speed!!).

Backup:

"backup_script.sh"  dirpath or filepath /dev/rmt0

- DONĀ“T COMPRESS  THE ORIGINAL SOURCE FILE(S) / DIRECTORIES , COMPRESSED FILES ONLY
IN TAPE. THE COMPRESS INTERMEDIARY PROCESS SHOULD  NOT CHANGE, AT ANY MOMENT,
THE ORIGINAL SOURCE FILE(S) OR DIRECTORY(IES) .

- DUPLICATE THE FILE FOR TEMPORARY JOB IS NOT DESIRABLE DUE TO DISK IO PERFORMANCE
IMPACT.

Shell Script 2 - restore a full directory recursivelly or specific file from the tape

Ex: "restore_script.sh"  dirpath or filepath  /dev/rmt0

- DIRECT File Restore without pre-restoring a big file container (Example: a full tar or compressed file)

restore 1 GB tar.Z file  to the disc for further restoring a single and small
file would be bad. So I imagine each file be compressed individually before writing to tape,
during the backup process,  instead generating a big compressed file with all files before
writing to the tape.

Compress tool can be gzip, pkzip, compress, bzip, etc.

Ex:


1- before backup

/home/dir1/file1
/home/dir1/file2
/home/dir2/file3
/home/dir2/dir3/file4
/home/dir2/dir4/file5

2 - backinp up

Backup examples:

a) "backup_script.sh"  /home/dir1 /dev/rmt0

b) "backup_script.sh"  /home/dir2/dir4/file5  /dev/rmt0


3 - After  backup (files on tape, JUST one example, is not necessary write files separately, it is just
my idea to improve the restore of individual files !!!)

a) /home/dir1/file1.Z
    /home/dir1/file2.Z

b) /home/dir2/dir4/file5.Z

4 - Restore examples

Ex:

a) "restore_script.sh"  /home/dir1 /dev/rmt0

b) "restore_script.sh"  /home/dir2/dir4/file5.Z  /dev/rmt0


Best Regards,
Artur Dietrich
Start your free trial to view this solution
Question Stats
Zone: OS
Question Asked By: artur_dietrich
Solution Provided By: JustUNIX
Participating Experts: 3
Solution Grade: B
Views: 79
Translate:
Loading Advertisement...
03.31.2008 at 10:47AM PDT, ID: 21247521

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
03.31.2008 at 01:49PM PDT, ID: 21249056

Rank: Wizard

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
04.02.2008 at 05:28AM PDT, ID: 21262272

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
04.06.2008 at 07:47PM PDT, ID: 21294070

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
04.07.2008 at 02:31AM PDT, ID: 21295334

Rank: Wizard

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
04.08.2008 at 01:15AM PDT, ID: 21303326

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
 
Loading Advertisement...
Microsoft
  • Internet Protocols
  • Applications
  • Development
  • OS
  • Hardware
  • Windows Security
Apple
  • Operating Systems
  • Hardware
  • Programming
  • Networking
  • Software
Internet
  • Search Engines
  • File Sharing
  • WebTrends / Stats
  • Spy / Ad Blockers
  • Web Browsers
  • New Net Users
  • Web Development
  • Chat / IM
  • Anti Spam
  • Web Servers
  • Anti-Virus
  • Email Clients
Gamers
  • Tips
  • Online / MMORPG
  • Puzzle
  • Emulators
  • Action / Adventure
  • Role Playing
  • Consoles
  • Game Programming
  • Strategy
  • Sports
  • Misc
  • Computer Games
Digital Living
  • Hardware
  • New Net Users
  • New Users
  • Software
  • Digital Music
  • Gaming World
  • Home Security
  • Apple
  • Networking Hardware
Virus & Spyware
  • Vulnerabilities
  • IDS
  • Encryption
  • Anti-Virus
  • Operating Systems Security
  • Software Firewalls
  • WebApplications
  • Cell Phones
  • Operating Systems
  • Internet
  • Hardware Firewalls
Hardware
  • Handhelds / PDAs
  • Displays / Monitors
  • Components
  • Networking Hardware
  • Peripherals
  • Laptops/Notebooks
  • Storage
  • Servers
  • Desktops
  • New Users
  • Misc
  • Apple
Software
  • System Utilities
  • Industry Specific
  • Network Management
  • Photos / Graphics
  • Page Layout
  • VMWare
  • Misc
  • Web Development
  • OS
  • CYGWIN
  • Voice Recognition
  • Message Queue
  • Quality Assurance
  • Security
  • Firewalls
  • MultiMedia Applications
  • Development
  • Database
  • Office / Productivity
  • Business Management
  • OS/2 Apps
  • Server Software
  • Internet / Email
ITPro
  • OS
  • Storage
  • Encryption
  • Operating Systems Security
  • Apple Hardware
  • Laptops & Notebooks
  • Servers
  • Networking Hardware
  • Peripherals
  • Devices
  • Displays / Monitors
  • WebTrends / Stats
  • Search Engines
  • Firewalls
  • WebApplications
  • IDS
  • Vulnerabilities
  • Email Clients
  • File Sharing
  • Spy / Ad Blockers
  • Web Browsers
  • Web Servers
  • Networking
  • Anti-Virus
  • Chat / IM
  • Anti Spam
Developer
  • Web Servers
  • Web Browsers
  • Game Programming
  • Dev Tools
  • Industry Specific
  • Office / Productivity
  • Database
  • CYGWIN
  • Web Development
  • Search Engines
  • File Sharing
  • WebTrends / Stats
  • Programming
  • Content Management
  • Application Servers
  • Protocols
Storage
  • Removable Backup Media
  • Storage Technology
  • Servers
  • Grid
  • Remote Access
  • Backup / Restore
  • Misc
  • Hard Drives
OS
  • Miscellaneous
  • Security
  • Development
  • Linux
  • VMWare
  • MainFrame OS
  • Unix
  • Apple
  • OS / 2
  • AS / 400
  • BeOS
  • Microsoft
  • VMS / OpenVMS
Database
  • Oracle
  • Miscellaneous
  • MySQL
  • Software
  • Sybase
  • Contact Management
  • PostgreSQL
  • Data Manipulation
  • Clarion
  • InterSystems Cache
  • Siebel
  • MUMPS
  • OLAP
  • SQLBase
  • SAS
  • GIS & GPS
  • 4GL
  • Berkeley DB
  • DB2
  • Informix
  • Interbase / Firebird
  • FoxPro
  • Reporting
  • LDAP
  • Filemaker Pro
  • MS SQL Server
  • dBase
  • MS Access
Security
  • Misc
  • Web Browsers
  • Software Firewalls
  • Operating Systems Security
  • File Sharing
  • Spy / Ad Blockers
  • Vulnerabilities
  • WebApplications
  • IDS
  • Anti-Virus
  • Encryption
  • Anti Spam
  • Email Clients
  • VPN
  • Chat / IM
Programming
  • Editors IDEs
  • Installation
  • Handhelds / PDAs
  • Multimedia Programming
  • System / Kernel
  • Algorithms
  • Game
  • Signal Processing
  • Project Management
  • Open Source
  • Database
  • Misc
  • Languages
  • Processor Platforms
  • Theory
Web Development
  • Scripting
  • Blogs
  • Web Servers
  • Software
  • Search Engines
  • Web Graphics
  • Images
  • Internet Marketing
  • Images and Photos
  • Components
  • Document Imaging
  • Web Languages/Standards
  • Illustration
  • WebApplications
  • Fonts
  • WebTrends / Stats
  • Authoring
  • Digital Camera Software
  • Miscellaneous
Networking
  • Protocols
  • Apple Networking
  • Network Management
  • Message Queue
  • Application Servers
  • Content Management
  • File Servers
  • Email Servers
  • Misc
  • Java Editors & IDEs
  • Wireless
  • Networking Hardware
  • Backup / Restore
  • System Utilities
  • ISPs & Hosting
  • Web Servers
  • Storage Technology
  • Removable Backup Media
  • Servers
  • Broadband
  • Grid
  • OS / 2
  • Novell Netware
  • Unix Networking
  • Windows Networking
  • Security
  • Telecommunications
  • Operating Systems
  • Linux Networking
Other
  • Community Advisor
  • Lounge
  • Community Support
  • New Net Users
  • Philosophy / Religion
  • Math / Science
  • Miscellaneous
  • URLs
  • Expert Lounge
  • Politics
  • Puzzles / Riddles
Community Support
  • Suggestions
  • New to EE
  • New Topics
  • Community Advisor
  • CleanUp
  • Announcements
  • General
  • Feedback
  • Input
  • EE Bugs
 
03.31.2008 at 10:47AM PDT, ID: 21247521
Right, 33GB/hour (~9MB/s) is nowhere near LTO3 peak performance (80MB/s).  But I would not waste the time on shell scripts -- if you cannot afford commercial backup software, use Bacula or Amanda.
 
03.31.2008 at 01:49PM PDT, ID: 21249056

Rank: Wizard

Years ago, I read a post that the optimum block size for a tape was 126 blocks (63K). I did a few experiments that seemed to confirm the that, so I've used it ever since. Using gzip (tar -czf) does get more onto the tape than hardware compression does, but there is no chance of tar recovering from a tape read error half way through if you use it.
I get a tad over 1MB/s reported by tar to my HP DDS-3, only a little over the  3.3GB / hour that you see
 
04.02.2008 at 05:28AM PDT, ID: 21262272
Using software compression will definitely slow down you backup as your
CPU will have to compress first, and send to tape secondly.

LTO3 accepts data quite quickly -- but you will have to make sure you send
it stuff fast enough! Most of the time, using a single tar or cpio stream will
not be enough to keep your drive streaming at good throughput rate.

You may try do use dd with a large block size for buffering (e.g. 64 MB):
# tar cvf - /my/dir | dd of=/my/tape/device bs=`expr 8192 \* 8192`k

Restore:
# dd if=/my/tape/device bs=`expr 8192 \* 8192`k | tar xvf -
 
04.06.2008 at 07:47PM PDT, ID: 21294070
JustUnix,

Tx for the idea but, how can I restore especific file or dir ?

Artur
 
04.07.2008 at 02:31AM PDT, ID: 21295334

Rank: Wizard

Just add the filename to dd if=/my/tape/device bs=`expr 8192 \* 8192`k | tar xvf -
 
04.08.2008 at 01:15AM PDT, ID: 21303326
To get a listing of backup contents:
  # dd if=/my/tape/device bs=`expr 8192 \* 8192`k | tar tvf -

And to restore a specific file (or directory) use
  # dd if=/my/tape/device bs=`expr 8192 \* 8192`k | tar xvf - my-dir/my-file
Make sure to use *exactly* the path name from your directory listing to
retrieve "my-dir/my-file"
Accepted Solution
 
 
20080236-EE-VQP-29 / EE_QW_2_20070628