Solved

Logging into server and copying files using perl

Posted on 2011-10-01
23
298 Views
Last Modified: 2012-05-12
Hi

Below is the script which is currently reading a log file and printing out the data, then it is doing SSH to a different server and going to a location

But can anyone please help me regarding an addition for this existing script.

After " ls-d" to that path, it has look for the filenames it is reading from the above file and copy them one by one to another folder called  FINISHED.

my $dir = '/usr/home/Files/Test_files';


# get the text files from the specified dir
opendir DIR, $dir or die "could not open directory $dir: $!";
my @files = grep /\.txt$/, readdir DIR;
closedir DIR;


foreach my $fil (@files) {
# read the ids from the files
    open IN, "$dir/$fil" or die "could not read $fil: $!";
       my @ids = map { "'$_'" } map { chomp; split /\s+/ } <IN>;
     print "checking ", scalar(@ids), " ids from $fil\n";
    print "@ids\n\n";
    

          
my $path4 = "/home/d/test/files";
          
my $tmp1 = "ssh prodadm\@173.12.196.10 'ls -d $path4'";
print "$path4\n";

}            

Open in new window


0
Comment
Question by:new_perl_user
  • 12
  • 11
23 Comments
 

Author Comment

by:new_perl_user
ID: 36904130
Can anyone please  look into this
0
 
LVL 9

Expert Comment

by:parparov
ID: 36904412
Can you provide examples of the process, like:
* Sample content of a txt file to read IDs from
* Sample content of the source directory before the move
* Sample content of the target directory after the move
0
 

Author Comment

by:new_perl_user
ID: 36904561
Sample content of the Text File:

DESNUM078 | 2554006S 
DESNUM078 | 65341080S 
DESNUM078 | 65720140S 
DESNUM075 | 65740220S 
DESNUM081 | 65750140S 
DESNUM080 | 66020070S 
DESNUM087 | 101558914 
DESNUM098 | 65650500S 

Open in new window


Sample content of the source Directory: "/home/d/test/files" (DESNUM.. is the Dir  &  65720140S( Sub Dir..)"
DESNUM075 (DIR)-> 457890345( Sub Directory..),  789024S (Sub Directory..)
DESNUM056 -> 12467890S, 56723456S

Open in new window


Sample content of the target directory: "/home/d/test/FINISHED"

DESNUM023 -> 4579890S,  45765890S
...

Open in new window



0
 
LVL 9

Expert Comment

by:parparov
ID: 36904795
I beg pardon, but the sample file and the sample contents of directories do not seem to have any relation between them, so the logic still escapes.

Do you need to copy the subdirs 2554006S, 65341080S, 65720140S from directory
/home/d/test/files/DESNUM078 to /home/d/test/FINISHED/DESNUM078 ?
Or we need to copy specific files?
0
 

Author Comment

by:new_perl_user
ID: 36904935

sorry I was just giving the sample, but not the matching information.

I am already reading the contents of the file,and next logging into the second server and Ls-d to the  Source Dir. Till it I am able to come up.

Next..
For example:
In the source dir it has to look  for DESNUM078 | 2554006S(first read from the log file) . If it finds DESNUM078 look for the subdir 2554006S .

Then make a copy this to the Destination.  If the top level DIR DESNUM078 exsists just copy the Sub Dir. If not copy both Parent and Child.

Then repeat the same for all until the last file in the log file.

0
 
LVL 9

Expert Comment

by:parparov
ID: 36905460
Try this code. The argument is the directory with the TXT files.
#!/usr/bin/perl

use strict;
use warnings;

my $txt_dir;
$txt_dir = shift || die "Usage: $0 txt_dir\n";
opendir(TXT_DIR, $txt_dir) || die "Can't read directory $txt_dir: $!\n";
my @txt_files = grep /\.txt$/, readdir TXT_DIR;
closedir TXT_DIR;

my %dirs = ();
for my $txt_file (@txt_files) {
	open(T, "$txt_dir/$txt_file");
	while (<T>) {
		my($dir, $subdir) = /^(\w+)\s+\|\s+(\w+)/;
		$dirs{$dir} ||= [];
		push(@{$dirs{$dir}}, $subdir);
	};
	close T;
}

my $src_base_dir = "/home/d/test/files";
my $dst_base_dir = "/home/d/test/FINISHED";
my $host = 'parparov@127.0.0.1';

DIR:
for my $dir (keys %dirs) {
	my $dst_dir = "$dst_base_dir/$dir";
	my $cmd = "ssh $host cd $dst_dir";
	system($cmd);
	if ($?) {
		print "Directory $dst_dir does not exist, creating...\n";
		my $cmd = "ssh $host mkdir -p $dst_dir";
		system($cmd);
		if ($?) {
			print "Directory $dst_dir could not be created: $!\n";
			next DIR;
		}
		print "Destination directory $dst_dir created.\n";
	}
	SUBDIR:
	for my $subdir (@{$dirs{$dir}}) {
		my $cmd;
		my $src_dir = "$src_base_dir/$dir/$subdir";
		$cmd = "ssh $host cd $src_dir";
		system($cmd);
		if ($?) {
			print "Error: no source directory $src_dir\n";
			next SUBDIR;
		}
		$cmd = "ssh $host cp -a $src_dir $dst_dir\n";
		system($cmd);
		if ($?) {
			print "Copy $src_dir to $dst_dir failed: $!\n";
			next SUBDIR;
		}
		print "Copied $src_dir to $dst_dir successfully\n";
	}
}

Open in new window

0
 

Author Comment

by:new_perl_user
ID: 36905845
Hi,

Thank you so much it is working but one error and one small addition.

Error:  while copying it is copying stuff but throwing out an error "Failed to Preserve ownership for /home/d/test/FINISHED/DESNUM078/2554006S/file1..

Addition: If possible can it send an email with the log file attached  after the copying is done or if it failed to copy or any problem with the execution.
0
 
LVL 9

Expert Comment

by:parparov
ID: 36905903
It's not exactly an error but more of a warning. It means that the owner of the original file is different, and you can't preserve this ownership if you're not doing it as root. If you will replace 'cp -a' with 'cp -r' you won't receive it as well.

As of the email, can you install perl modules on the computer running the script?
0
 

Author Comment

by:new_perl_user
ID: 36905926
Yes, I can install and some of them are already there like MIME::LITE
0
 
LVL 9

Expert Comment

by:parparov
ID: 36906025
Here is a modified version to accommodate your latest requests
#!/usr/bin/perl

use strict;
use warnings;

use MIME::Lite;

my $txt_dir;
$txt_dir = shift || die "Usage: $0 txt_dir\n";
opendir(TXT_DIR, $txt_dir) || die "Can't read directory $txt_dir: $!\n";
my @txt_files = grep /\.txt$/, readdir TXT_DIR;
closedir TXT_DIR;

my %dirs = ();
for my $txt_file (@txt_files) {
	open(T, "$txt_dir/$txt_file");
	while (<T>) {
		my($dir, $subdir) = /^(\w+)\s+\|\s+(\w+)/;
		$dirs{$dir} ||= [];
		push(@{$dirs{$dir}}, $subdir);
	};
	close T;
}

my $src_base_dir = "/home/d/test/files";
my $dst_base_dir = "/home/d/test/FINISHED";
my $host = 'parparov@127.0.0.1';
our @INFO = ();

sub my_debug ($) {

	my $message = $_;
	print $message;
	push(@INFO, $message);
}

DIR:
for my $dir (keys %dirs) {
	my $dst_dir = "$dst_base_dir/$dir";
	my $cmd = "ssh $host cd $dst_dir";
	system($cmd);
	if ($?) {
		print "Directory $dst_dir does not exist, creating...\n";
		my $cmd = "ssh $host mkdir -p $dst_dir";
		system($cmd);
		if ($?) {
			my_debug "Directory $dst_dir could not be created: $!\n";
			next DIR;
		}
		print "Destination directory $dst_dir created.\n";
	}
	SUBDIR:
	for my $subdir (@{$dirs{$dir}}) {
		my $cmd;
		my $src_dir = "$src_base_dir/$dir/$subdir";
		$cmd = "ssh $host cd $src_dir";
		system($cmd);
		if ($?) {
			my_debug "Error: no source directory $src_dir\n";
			next SUBDIR;
		}
		$cmd = "ssh $host cp -r $src_dir $dst_dir\n";
		system($cmd);
		if ($?) {
			my_debug "Copy $src_dir to $dst_dir failed: $!\n";
			next SUBDIR;
		}
		print "Copied $src_dir to $dst_dir successfully\n";
	}
}

my $msg = MIME::Lite->new(
	From     =>'copying-reporter@yourserver.com',
	To       =>'you@youraddress.com',
	Subject  => @INFO ? "Errors in copying" : "Copying completed OK",
	Data     => @INFO ? join("", @INFO) : "No errors were detected",
);

$msg->send;

Open in new window

0
 
LVL 9

Accepted Solution

by:
parparov earned 500 total points
ID: 36906040
oops, a correction required - here it is.
#!/usr/bin/perl

use strict;
use warnings;

use MIME::Lite;

my $txt_dir;
$txt_dir = shift || die "Usage: $0 txt_dir\n";
opendir(TXT_DIR, $txt_dir) || die "Can't read directory $txt_dir: $!\n";
my @txt_files = grep /\.txt$/, readdir TXT_DIR;
closedir TXT_DIR;

my %dirs = ();
for my $txt_file (@txt_files) {
        open(T, "$txt_dir/$txt_file");
        while (<T>) {
                my($dir, $subdir) = /^(\w+)\s+\|\s+(\w+)/;
                $dirs{$dir} ||= [];
                push(@{$dirs{$dir}}, $subdir);
        };
        close T;
}

my $src_base_dir = "/home/d/test/files";
my $dst_base_dir = "/home/d/test/FINISHED";
my $host = 'parparov@127.0.0.1';
our @INFO = ();

sub my_debug ($) {

        my $message = shift;
        print $message;
        push(@INFO, $message);
}

DIR:
for my $dir (keys %dirs) {
        my $dst_dir = "$dst_base_dir/$dir";
        my $cmd = "ssh $host cd $dst_dir";
        system($cmd);
        if ($?) {
                print "Directory $dst_dir does not exist, creating...\n";
                my $cmd = "ssh $host mkdir -p $dst_dir";
                system($cmd);
                if ($?) {
                        my_debug "Directory $dst_dir could not be created: $!\n";
                        next DIR;
                }
                print "Destination directory $dst_dir created.\n";
        }
        SUBDIR:
        for my $subdir (@{$dirs{$dir}}) {
                my $cmd;
                my $src_dir = "$src_base_dir/$dir/$subdir";
                $cmd = "ssh $host cd $src_dir";
                system($cmd);
                if ($?) {
                        my_debug "Error: no source directory $src_dir\n";
                        next SUBDIR;
                }
                $cmd = "ssh $host cp -r $src_dir $dst_dir\n";
                system($cmd);
                if ($?) {
                        my_debug "Copy $src_dir to $dst_dir failed: $!\n";
                        next SUBDIR;
                }
                print "Copied $src_dir to $dst_dir successfully\n";
        }
}

my $msg = MIME::Lite->new(
        From     =>'copying-reporter@yourserver.com',
        To       =>'you@youraddress.com',
        Subject  => @INFO ? "Errors in copying" : "Copying completed OK",
        Data     => @INFO ? join("", @INFO) : "No errors were detected",
);

$msg->send;

Open in new window

0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:new_perl_user
ID: 36906214
Hi,
Thank  a lot  for the above code it is working, but it is also saying "Copying Completed ok" even if it did not copy anything. I mean when there are no files to copy.

I just gave an empty file as input  to check... It should say Errors in Copying right
0
 
LVL 9

Expert Comment

by:parparov
ID: 36906352
I don't know. There was no explicit specification for that behaviour.

To comply with it, put in line 36:
if (! keys %dirs) {print "Nothing to copy\n"; exit; }

Open in new window

0
 

Author Comment

by:new_perl_user
ID: 36906471
so can the Subject in the email take only two conditions  because I was trying like the below so that I can get the message in email.

if (! keys %dirs) {my_debug "Nothing to copy\n"; exit; }--- line 36

my $msg = MIME::Lite->new(
        From     =>'copying-reporter@yourserver.com',
        To       =>'you@youraddress.com',
        Subject  => @INFO ? " Nothing to Copy" : "Errors in copying" : "Copying completed OK",
        Data     => @INFO ? join("", @INFO) : "No errors were detected",
);

$msg->send;

Open in new window


But it is throwing out Error at the Subject.
0
 
LVL 9

Expert Comment

by:parparov
ID: 36907096
It's a bit uglier to use ternary conditional there, but we could do as following:
at line 36 do
my $nothing_to_copy = 0;
if (! keys %dirs) {print "Nothing to copy\n"; $nothing_to_copy = 1; goto MAIL; }

Open in new window

The MIME::Lite call should change then to:
MAIL:
my $msg = MIME::Lite->new(
        From     =>'copying-reporter@yourserver.com',
        To       =>'you@youraddress.com',
        Subject  => $nothing_to_copy ? "Nothing to Copy" : @INFO ? "Errors in copying" : "Copying completed OK",
        Data     => $nothing_to_copy ? "Nothing to Copy" : @INFO ? join("", @INFO) : "No errors were detected",
);

$msg->send;

Open in new window

0
 

Author Comment

by:new_perl_user
ID: 36907132
Thanks a ton for all the help.  And finally Two questions:)

1) Is there a way to assign more points than 500  for this question.
2) In the script we are grepping .txt files, Is there a way we can grep file by date because after posting the above requirement I came to know that the filename they supply to me would be in the format.

Everday they are going to give me a new file in the below format and also the old ones  will still be there
 So I need to grep only today's file from that bunch of old files.. All files will be in the below format..

"test_20111003_HDF.txt"
0
 
LVL 9

Expert Comment

by:parparov
ID: 36907157
Regarding 500 points, you'll have to ask the admin
We can grep by date, but that would needs to be a carefully formulated separate question.
For instance, whether we should look into real file's date, or the one contained in its name.
0
 

Author Comment

by:new_perl_user
ID: 36907201
ok. I will ask the admin regarding the points and I will open another question regarding the date of the file.
0
 

Author Comment

by:new_perl_user
ID: 36907225
Hi,

Opened up a new question "Grep a file by Date". Can you pls let me know the workaround for it..

Thanks,
0
 
LVL 9

Expert Comment

by:parparov
ID: 36913443
hope I answered it.
0
 

Author Comment

by:new_perl_user
ID: 36926893
Hi,

Yes, it is working for grepping a file by date and I accepted it.

But for the above script , is it possible to generate a log file  for the list it is copying  and send as an attachment  in email



0
 
LVL 9

Expert Comment

by:parparov
ID: 36933451
Yes, but I think it deserves a separate question. :)
0
 

Author Comment

by:new_perl_user
ID: 36958878
Thank you and I will open it uo. Spoke with the admin to check if i can assign more points but unfortunately they said no. I am sorry.
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Perl Tutorial 6 127
perl split by | 2 81
Check file date before getting folder path 1 56
Bash one liner, start and end time calculation 13 95
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now