[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 310
  • Last Modified:

Read CSV file from remote servers and combine into 1

Hey experts,

I have a bunch of hosts all collecting home directory info on a specific user and creating a csv file saved locally on the respective host.

I'm writing a perl script to collect the remote data from each file and append it to one large csv file. This seems to be a little complicated to me because I'm trying to add headers into the file to distinguish from which host the data is coming from.

The remote file looks like this:

***** today's date *********
***** heaviest dirs/files ***

100MB, /path/to/dir/
70MB, /path/to/dir/file1
10MB, /path/to/dir/file2
1.5MB, /path/to/dir/dir2
500K, /path/to/dir/dir2/otherfile
etc....

To explain what I tried to do:
 - open local output file
 - loop through hostname list and ssh into hosts
 - read remote csv (ignoring first 3 lines)
 - print header into local output file to distinguish origin of data
 - somehow read the csv and output to local file ???

This is the part where I'm sort of stuck. Actually, I'm pretty sure the code is all wrong cause while testing it, it hung. I think it has to do something with the shell it's opening on the remote host...

This has been racking my brain for a week now. If I could give more than 500 points I would...

open (FILE2, "> /home/user/du_collection.csv") or die "Cannot create the csv file";
my @output;

for(my $i = 0; $i <= $#list; $i++){
        my @ssh = `ssh $user\@$list[$i]`;
        open (FILE1, "//home/foobar/du_out.csv") or die "Cannot open the csv file";
        print FILE2 "************ $list[$i] ************";
        <FILE1>;<FILE1>;<FILE1>;     # ignore first 3 lines of csv file
        while (<FILE1>){
                foreach my $line (<FILE1>){
                        
                       [ commands to read csv and print to FILE2 ]
                        
                         }
                close FILE1;
                 }
}

close FILE2;

Open in new window


Any and all help is appreciated. Thanks!
0
akatsuki27
Asked:
akatsuki27
  • 6
  • 5
  • 5
1 Solution
 
jeromeeCommented:
Hi akatsuki27,
How do you get the data from the remote hosts?
I do see your call to ssh but I fail to see which command you are running on the remote host to get the data you need.
Do you run "du"?
0
 
FishMongerCommented:
Your ssh command isn't doing what you think and the "open (FILE1,..." line is not accessing the remote file.

Use the File::Remote module to access the remote files.

This is untested and may need some minor tweaking.
use File::Remote;

my $csv = '/home/user/du_collection.csv';
open (my $csv_fh, '>', $csv) or die "Cannot create <$csv> file $!";

for my $host ( @list ){

    chomp $host;
    open (my $remote_fh, "$host:/home/foobar/du_out.csv")
      or warn "Cannot open the csv file on $host $!\n" and next;

    <$remote_fh> for (1..3);
    
    print $csv_fh "************ $host ************";

    while (<$remote_fh>){
        print { $csv_fh } $_;
    }
    close $remote_fh;
}

close $csv_fh;

Open in new window

0
 
akatsuki27Author Commented:
Jeromee,

I have a script on the remote box that runs du and stores the output in a csv file. The script I'm working on now should connect to each remote host and gather the disk usage data and store it in 1 file.

FishMonger,

Is that module available for RHEL? Yum doesn't find it. Is there another module that will allow me to handle remote files?
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
akatsuki27Author Commented:
Also FishMonger can you explain why the ssh is not working?
0
 
jeromeeCommented:
Hi akatsuki27,
Is ftp/sftp an option to download the files?
0
 
akatsuki27Author Commented:
Unfortunately no. I'm limited to ssh or scp. ftp is not set up on any/all internal host.
0
 
jeromeeCommented:
scp will do too or you could run ssh to "cat" the file.
Check out http://search.cpan.org/~ivan/Net-SCP-0.08/SCP.pm for scp
0
 
akatsuki27Author Commented:
I think reading the file once and processing will be my best choice. Since the home directories I'm checking get really huge. I would rather not copy those files over from 50+ hosts.

Do you know of the module FishMonger mentioned? I dont find it using yum. Is there another module that does similar things?
0
 
FishMongerCommented:
0
 
FishMongerCommented:
This is the command I normally use to install modules.

perl -MCPAN -e 'install File::Remote'
0
 
jeromeeCommented:
If you don't want to download all files, then you will need to install and run a script on all 50+ hosts.
0
 
akatsuki27Author Commented:
Jeromee,

I do have a script on all 50+ hosts running du. I still need a way to collect that data into 1 csv for general use.
0
 
jeromeeCommented:
Hi akatsuki27,
So you do need to download 50+ files once they have been generated on each individual machine/host.
I would recommend that you use scp (or a Perl wrapper around it as suggested earlier with http://search.cpan.org/~ivan/Net-SCP-0.08/SCP.pm)
Use Fishmonger excellent pointers on how to download any Perl module you want.
Or just use plain scp.

Good luck.
0
 
FishMongerCommented:
The file::Remote module that I suggested uses scp under the hood, which simplifies the process because you don't need to manually scp the file.

The example I gave will need a couple minor adjustments.  One being that you'll need to change:
use File::Remote;

Open in new window

to:
use File::Remote qw(:replace);

Open in new window

0
 
akatsuki27Author Commented:
FishMonger

This line is so it can ignore the first 3 lines of the csv file on the remote host?

<$remote_fh> for (1..3);

Open in new window

0
 
FishMongerCommented:
yes, that's correct.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

  • 6
  • 5
  • 5
Tackle projects and never again get stuck behind a technical roadblock.
Join Now