Solved

Compress and ship log files

Posted on 2011-09-26
9
469 Views
Last Modified: 2012-05-12
Servers are running SQL 2008 STD. I need to compress the log files created by LSBACKUP job and ship them over WAN to backup (standby) server, then extract them for LSRESTORE to do it's thing.  Standard LSCOPY job won't do the job because some of the trn files end up 20GB in size.  However, they zip or rar down to around 100mb, which would transport much better!  I need an automated solution, or a looping batch. Send server needs to only copy new files, and recipent end needs to ignore files already extracted.  I hope to be able to set up regular log shipping, and then just disable the lscopy job.  

Have looked at Unison file transfer, which supports compress before copy, but it could delete the recipient copy before lsrestore job applies it.  Is there a command line for winrar?  Need some ideas and inspiration here.
0
Comment
Question by:billherde
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
9 Comments
 
LVL 10

Expert Comment

by:Arman Khodabande
ID: 36707871
Hi
You can make a batch file with winrar command line options.
I attached the command line options. I know you can make something out of it. !
I'm busy these days . . .
rar-syntax.txt
0
 
LVL 3

Author Comment

by:billherde
ID: 36708178
I have the syntax for rar and unrar, what I am lacking is a part that will make a seperate file for each file found in the directory. It would be nice if they matched the filename being compressed, but that is not needed either.  I pretty much got the command line to compress and copy the file looking for archive bit and then reset archive bit on completion. Extraction and deleting the file at the destination works with if-then-else, but I don't know how to get a filelist into a table of variables and use it.
0
 
LVL 10

Accepted Solution

by:
Arman Khodabande earned 500 total points
ID: 36710341
Put the following script in a batch file and it adds files to separate archives. Just change the paths in script to be what you need on your system.
setlocal
set sourcepath=c:\<path to folder to backup>
set destpath=c:\<path to folder to store backups>
set wrpath=C:\Program Files\WinRAR
if NOT EXIST %destpath% md %_destpath%
for %%I in (%sourcepath%\*.*) do "%wrpath%\winrar" A "%destpath%\%%~nI.rar" "%%I"

Open in new window


Do your files in source path get deleted everyday or they're accumulative?
0
How Do You Stack Up Against Your Peers?

With today’s modern enterprise so dependent on digital infrastructures, the impact of major incidents has increased dramatically. Grab the report now to gain insight into how your organization ranks against your peers and learn best-in-class strategies to resolve incidents.

 
LVL 3

Author Comment

by:billherde
ID: 36710493
They are deleted by the maintenance job after 72 hours.  Thank You! that is the part I needed.
0
 
LVL 3

Author Closing Comment

by:billherde
ID: 36710504
Also found a canned program called Watchdirectory that does the job, but is limited to Zip compression.
0
 
LVL 10

Expert Comment

by:Arman Khodabande
ID: 36710806
Good to hear that.
But zip files may get corrupted during shipping!
Plus that you can compress the files further with Winrar by choosing high compression levels. (This makes the transfer faster too because of small files)
Good luck
0
 
LVL 3

Author Comment

by:billherde
ID: 36710907
Agreed.  That is why I am looking to scripting this using rar and unrar.  Initial testing found a 15gb tlog file compressed to around 1gb with winzip, but squeezed down to 125mb with winrar. My WAN link to the backup server is limited to 10mbps and offsite backup is out there as well. Watchdirectory has some desirable history logging, and will run as a service, but seems a bit much for a one trick solution.
0
 
LVL 10

Expert Comment

by:Arman Khodabande
ID: 36711128
Right.
Do you use -m5 switch to maximize the compression ratio? Use best compression method (slightly more compressive, but slowest) to reach the maximum file compression and reduced size.
0
 
LVL 3

Author Comment

by:billherde
ID: 36711578
Yes.  Makes a big difference for the small set!  After all, size does matter!
0

Featured Post

Guide to Performance: Optimization & Monitoring

Nowadays, monitoring is a mixture of tools, systems, and codes—making it a very complex process. And with this complexity, comes variables for failure. Get DZone’s new Guide to Performance to learn how to proactively find these variables and solve them before a disruption occurs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Redirected folders in a windows domain can be quite useful for a number of reasons, one of them being that with redirected application data, you can give users more seamless experience when logging into different workstations.  For example, if a use…
You might have come across a situation when you have Exchange 2013 server in two different sites (Production and DR). After adding the Database copy in ECP console it displays Database copy status unknown for the DR exchange server. Issue is strange…
In this video, we discuss why the need for additional vertical screen space has become more important in recent years, namely, due to the transition in the marketplace of 4x3 computer screens to 16x9 and 16x10 screens (so-called widescreen format). …
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question