Solved

gzip results of remote find and gunzip locally in mone command

Posted on 2011-03-14
10
659 Views
Last Modified: 2013-11-05
Hi,

I'm trying to pipe the results of a find command on a remote server through gzip to compress them and then gunzip them on a local server in a one-liner, if possible:

something like:
ssh remotesite "find /archive/userfiles/ -type f -mtime -1 | cat {} gzip -9 | gunzip > /local/folder"

Open in new window


Cheers
0
Comment
Question by:bwgexperts
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 2
10 Comments
 
LVL 31

Expert Comment

by:farzanj
ID: 35133434
You cannot do file transfer in one liner.  You either need to cat it and this way you may save it

ssh remotesite "find /archive/userfiles/ -type f -mtime -1 -exec  cat {} " > /local/folder/file




0
 

Author Comment

by:bwgexperts
ID: 35133494
Thanks for the reply, but this gut seems to think you can gzip & gunzip in the once command

http://www.commandlinefu.com/commands/view/4859/scp-with-compression.

What I need to do is plug in a find command into this structure.

Thanks again.
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35133550
Ok,

Try this then
 
ssh remotesite "find /archive/userfiles/ -type f -mtime -1 -exec  gunzip -c1 {} " | gunzip -c > /local/folder/file

Open in new window

0
NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

 
LVL 31

Assisted Solution

by:farzanj
farzanj earned 167 total points
ID: 35133560
Or this one.

I left ; in the above one

ssh remotesite "find /archive/userfiles/ -type f -mtime -1 -exec  cat {} \; | gunzip -c1 {} " | gunzip -c > /local/folder/file
0
 
LVL 12

Assisted Solution

by:mccracky
mccracky earned 83 total points
ID: 35133635
I don't think you can do it in a "one-liner" and you don't really need to gzip and gunzip it as ssh has compression built in.  In two lines you could:

1.  ssh -fC user@remotesite "find /archive/userfiles/ -type f -mtime -1 > ~/filelist
2.  scp -C user@remotesite:filelist /local/folder

The "-C" option of ssh and scp turns on gzip compression for the transfers.
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35133671
The example that you had quoted had only one file to be transferred.  I did not follow the same logic but for more files, it may be harder to do it in a single line.  As I said before, first you may need two commands as also shown by mccracky.
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35134135
Your DNS is not configured right.

I could not find your MX record.


dig mx www.mxtoolbox.com

; <<>> DiG 9.7.0-P2-RedHat-9.7.0-5.P2.el6 <<>> mx www.mxtoolbox.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17720
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;www.mxtoolbox.com.            IN      MX

;; AUTHORITY SECTION:
mxtoolbox.com.            5      IN      SOA      ns1.zoneedit.com. soacontact.zoneedit.com. 1133805690 14400 7200 1073741824 3600

;; Query time: 31 msec
;; SERVER: 192.168.2.2#53(192.168.2.2)
;; WHEN: Mon Mar 14 20:05:27 2011
;; MSG SIZE  rcvd: 95


FOR EXAMPLE

dig mx juno.com

; <<>> DiG 9.7.0-P2-RedHat-9.7.0-5.P2.el6 <<>> mx juno.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8211
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;juno.com.                  IN      MX

;; ANSWER SECTION:
juno.com.            5      IN      MX      10 mx.dca.untd.com.
juno.com.            5      IN      MX      10 mx.vgs.untd.com.

;; ADDITIONAL SECTION:
mx.vgs.untd.com.      5      IN      A      64.136.52.37
mx.dca.untd.com.      5      IN      A      64.136.44.37

;; Query time: 27 msec
;; SERVER: 192.168.2.2#53(192.168.2.2)
;; WHEN: Mon Mar 14 20:07:32 2011
;; MSG SIZE  rcvd: 109
0
 
LVL 31

Accepted Solution

by:
farzanj earned 167 total points
ID: 35134143
I send a mail on your root account. I used IP address.  Did you get any mail on your root account?

How to check?

Just type mail after becoming root
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35134149
Please check using a local machine.  Otherwise it would take for ever to troubleshoot.  This question relates to EMAIL not troubleshooting the entire network.
0
 

Author Closing Comment

by:bwgexperts
ID: 36452779
Cheers.
0

Featured Post

Secure Your WordPress Site: 5 Essential Approaches

WordPress is the web's most popular CMS, but its dominance also makes it a target for attackers. Our eBook will show you how to:

Prevent costly exploits of core and plugin vulnerabilities
Repel automated attacks
Lock down your dashboard, secure your code, and protect your users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The purpose of this article is to demonstrate how we can use conditional statements using Python.
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question