• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 321
  • Last Modified:

Logging into server and copying files using perl

Hi

Below is the script which is currently reading a log file and printing out the data, then it is doing SSH to a different server and going to a location

But can anyone please help me regarding an addition for this existing script.

After " ls-d" to that path, it has look for the filenames it is reading from the above file and copy them one by one to another folder called  FINISHED.

my $dir = '/usr/home/Files/Test_files';


# get the text files from the specified dir
opendir DIR, $dir or die "could not open directory $dir: $!";
my @files = grep /\.txt$/, readdir DIR;
closedir DIR;


foreach my $fil (@files) {
# read the ids from the files
    open IN, "$dir/$fil" or die "could not read $fil: $!";
       my @ids = map { "'$_'" } map { chomp; split /\s+/ } <IN>;
     print "checking ", scalar(@ids), " ids from $fil\n";
    print "@ids\n\n";
    

          
my $path4 = "/home/d/test/files";
          
my $tmp1 = "ssh prodadm\@173.12.196.10 'ls -d $path4'";
print "$path4\n";

}            

Open in new window


0
new_perl_user
Asked:
new_perl_user
  • 12
  • 11
1 Solution
 
new_perl_userAuthor Commented:
Can anyone please  look into this
0
 
parparovCommented:
Can you provide examples of the process, like:
* Sample content of a txt file to read IDs from
* Sample content of the source directory before the move
* Sample content of the target directory after the move
0
 
new_perl_userAuthor Commented:
Sample content of the Text File:

DESNUM078 | 2554006S 
DESNUM078 | 65341080S 
DESNUM078 | 65720140S 
DESNUM075 | 65740220S 
DESNUM081 | 65750140S 
DESNUM080 | 66020070S 
DESNUM087 | 101558914 
DESNUM098 | 65650500S 

Open in new window


Sample content of the source Directory: "/home/d/test/files" (DESNUM.. is the Dir  &  65720140S( Sub Dir..)"
DESNUM075 (DIR)-> 457890345( Sub Directory..),  789024S (Sub Directory..)
DESNUM056 -> 12467890S, 56723456S

Open in new window


Sample content of the target directory: "/home/d/test/FINISHED"

DESNUM023 -> 4579890S,  45765890S
...

Open in new window



0
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

 
parparovCommented:
I beg pardon, but the sample file and the sample contents of directories do not seem to have any relation between them, so the logic still escapes.

Do you need to copy the subdirs 2554006S, 65341080S, 65720140S from directory
/home/d/test/files/DESNUM078 to /home/d/test/FINISHED/DESNUM078 ?
Or we need to copy specific files?
0
 
new_perl_userAuthor Commented:

sorry I was just giving the sample, but not the matching information.

I am already reading the contents of the file,and next logging into the second server and Ls-d to the  Source Dir. Till it I am able to come up.

Next..
For example:
In the source dir it has to look  for DESNUM078 | 2554006S(first read from the log file) . If it finds DESNUM078 look for the subdir 2554006S .

Then make a copy this to the Destination.  If the top level DIR DESNUM078 exsists just copy the Sub Dir. If not copy both Parent and Child.

Then repeat the same for all until the last file in the log file.

0
 
parparovCommented:
Try this code. The argument is the directory with the TXT files.
#!/usr/bin/perl

use strict;
use warnings;

my $txt_dir;
$txt_dir = shift || die "Usage: $0 txt_dir\n";
opendir(TXT_DIR, $txt_dir) || die "Can't read directory $txt_dir: $!\n";
my @txt_files = grep /\.txt$/, readdir TXT_DIR;
closedir TXT_DIR;

my %dirs = ();
for my $txt_file (@txt_files) {
	open(T, "$txt_dir/$txt_file");
	while (<T>) {
		my($dir, $subdir) = /^(\w+)\s+\|\s+(\w+)/;
		$dirs{$dir} ||= [];
		push(@{$dirs{$dir}}, $subdir);
	};
	close T;
}

my $src_base_dir = "/home/d/test/files";
my $dst_base_dir = "/home/d/test/FINISHED";
my $host = 'parparov@127.0.0.1';

DIR:
for my $dir (keys %dirs) {
	my $dst_dir = "$dst_base_dir/$dir";
	my $cmd = "ssh $host cd $dst_dir";
	system($cmd);
	if ($?) {
		print "Directory $dst_dir does not exist, creating...\n";
		my $cmd = "ssh $host mkdir -p $dst_dir";
		system($cmd);
		if ($?) {
			print "Directory $dst_dir could not be created: $!\n";
			next DIR;
		}
		print "Destination directory $dst_dir created.\n";
	}
	SUBDIR:
	for my $subdir (@{$dirs{$dir}}) {
		my $cmd;
		my $src_dir = "$src_base_dir/$dir/$subdir";
		$cmd = "ssh $host cd $src_dir";
		system($cmd);
		if ($?) {
			print "Error: no source directory $src_dir\n";
			next SUBDIR;
		}
		$cmd = "ssh $host cp -a $src_dir $dst_dir\n";
		system($cmd);
		if ($?) {
			print "Copy $src_dir to $dst_dir failed: $!\n";
			next SUBDIR;
		}
		print "Copied $src_dir to $dst_dir successfully\n";
	}
}

Open in new window

0
 
new_perl_userAuthor Commented:
Hi,

Thank you so much it is working but one error and one small addition.

Error:  while copying it is copying stuff but throwing out an error "Failed to Preserve ownership for /home/d/test/FINISHED/DESNUM078/2554006S/file1..

Addition: If possible can it send an email with the log file attached  after the copying is done or if it failed to copy or any problem with the execution.
0
 
parparovCommented:
It's not exactly an error but more of a warning. It means that the owner of the original file is different, and you can't preserve this ownership if you're not doing it as root. If you will replace 'cp -a' with 'cp -r' you won't receive it as well.

As of the email, can you install perl modules on the computer running the script?
0
 
new_perl_userAuthor Commented:
Yes, I can install and some of them are already there like MIME::LITE
0
 
parparovCommented:
Here is a modified version to accommodate your latest requests
#!/usr/bin/perl

use strict;
use warnings;

use MIME::Lite;

my $txt_dir;
$txt_dir = shift || die "Usage: $0 txt_dir\n";
opendir(TXT_DIR, $txt_dir) || die "Can't read directory $txt_dir: $!\n";
my @txt_files = grep /\.txt$/, readdir TXT_DIR;
closedir TXT_DIR;

my %dirs = ();
for my $txt_file (@txt_files) {
	open(T, "$txt_dir/$txt_file");
	while (<T>) {
		my($dir, $subdir) = /^(\w+)\s+\|\s+(\w+)/;
		$dirs{$dir} ||= [];
		push(@{$dirs{$dir}}, $subdir);
	};
	close T;
}

my $src_base_dir = "/home/d/test/files";
my $dst_base_dir = "/home/d/test/FINISHED";
my $host = 'parparov@127.0.0.1';
our @INFO = ();

sub my_debug ($) {

	my $message = $_;
	print $message;
	push(@INFO, $message);
}

DIR:
for my $dir (keys %dirs) {
	my $dst_dir = "$dst_base_dir/$dir";
	my $cmd = "ssh $host cd $dst_dir";
	system($cmd);
	if ($?) {
		print "Directory $dst_dir does not exist, creating...\n";
		my $cmd = "ssh $host mkdir -p $dst_dir";
		system($cmd);
		if ($?) {
			my_debug "Directory $dst_dir could not be created: $!\n";
			next DIR;
		}
		print "Destination directory $dst_dir created.\n";
	}
	SUBDIR:
	for my $subdir (@{$dirs{$dir}}) {
		my $cmd;
		my $src_dir = "$src_base_dir/$dir/$subdir";
		$cmd = "ssh $host cd $src_dir";
		system($cmd);
		if ($?) {
			my_debug "Error: no source directory $src_dir\n";
			next SUBDIR;
		}
		$cmd = "ssh $host cp -r $src_dir $dst_dir\n";
		system($cmd);
		if ($?) {
			my_debug "Copy $src_dir to $dst_dir failed: $!\n";
			next SUBDIR;
		}
		print "Copied $src_dir to $dst_dir successfully\n";
	}
}

my $msg = MIME::Lite->new(
	From     =>'copying-reporter@yourserver.com',
	To       =>'you@youraddress.com',
	Subject  => @INFO ? "Errors in copying" : "Copying completed OK",
	Data     => @INFO ? join("", @INFO) : "No errors were detected",
);

$msg->send;

Open in new window

0
 
parparovCommented:
oops, a correction required - here it is.
#!/usr/bin/perl

use strict;
use warnings;

use MIME::Lite;

my $txt_dir;
$txt_dir = shift || die "Usage: $0 txt_dir\n";
opendir(TXT_DIR, $txt_dir) || die "Can't read directory $txt_dir: $!\n";
my @txt_files = grep /\.txt$/, readdir TXT_DIR;
closedir TXT_DIR;

my %dirs = ();
for my $txt_file (@txt_files) {
        open(T, "$txt_dir/$txt_file");
        while (<T>) {
                my($dir, $subdir) = /^(\w+)\s+\|\s+(\w+)/;
                $dirs{$dir} ||= [];
                push(@{$dirs{$dir}}, $subdir);
        };
        close T;
}

my $src_base_dir = "/home/d/test/files";
my $dst_base_dir = "/home/d/test/FINISHED";
my $host = 'parparov@127.0.0.1';
our @INFO = ();

sub my_debug ($) {

        my $message = shift;
        print $message;
        push(@INFO, $message);
}

DIR:
for my $dir (keys %dirs) {
        my $dst_dir = "$dst_base_dir/$dir";
        my $cmd = "ssh $host cd $dst_dir";
        system($cmd);
        if ($?) {
                print "Directory $dst_dir does not exist, creating...\n";
                my $cmd = "ssh $host mkdir -p $dst_dir";
                system($cmd);
                if ($?) {
                        my_debug "Directory $dst_dir could not be created: $!\n";
                        next DIR;
                }
                print "Destination directory $dst_dir created.\n";
        }
        SUBDIR:
        for my $subdir (@{$dirs{$dir}}) {
                my $cmd;
                my $src_dir = "$src_base_dir/$dir/$subdir";
                $cmd = "ssh $host cd $src_dir";
                system($cmd);
                if ($?) {
                        my_debug "Error: no source directory $src_dir\n";
                        next SUBDIR;
                }
                $cmd = "ssh $host cp -r $src_dir $dst_dir\n";
                system($cmd);
                if ($?) {
                        my_debug "Copy $src_dir to $dst_dir failed: $!\n";
                        next SUBDIR;
                }
                print "Copied $src_dir to $dst_dir successfully\n";
        }
}

my $msg = MIME::Lite->new(
        From     =>'copying-reporter@yourserver.com',
        To       =>'you@youraddress.com',
        Subject  => @INFO ? "Errors in copying" : "Copying completed OK",
        Data     => @INFO ? join("", @INFO) : "No errors were detected",
);

$msg->send;

Open in new window

0
 
new_perl_userAuthor Commented:
Hi,
Thank  a lot  for the above code it is working, but it is also saying "Copying Completed ok" even if it did not copy anything. I mean when there are no files to copy.

I just gave an empty file as input  to check... It should say Errors in Copying right
0
 
parparovCommented:
I don't know. There was no explicit specification for that behaviour.

To comply with it, put in line 36:
if (! keys %dirs) {print "Nothing to copy\n"; exit; }

Open in new window

0
 
new_perl_userAuthor Commented:
so can the Subject in the email take only two conditions  because I was trying like the below so that I can get the message in email.

if (! keys %dirs) {my_debug "Nothing to copy\n"; exit; }--- line 36

my $msg = MIME::Lite->new(
        From     =>'copying-reporter@yourserver.com',
        To       =>'you@youraddress.com',
        Subject  => @INFO ? " Nothing to Copy" : "Errors in copying" : "Copying completed OK",
        Data     => @INFO ? join("", @INFO) : "No errors were detected",
);

$msg->send;

Open in new window


But it is throwing out Error at the Subject.
0
 
parparovCommented:
It's a bit uglier to use ternary conditional there, but we could do as following:
at line 36 do
my $nothing_to_copy = 0;
if (! keys %dirs) {print "Nothing to copy\n"; $nothing_to_copy = 1; goto MAIL; }

Open in new window

The MIME::Lite call should change then to:
MAIL:
my $msg = MIME::Lite->new(
        From     =>'copying-reporter@yourserver.com',
        To       =>'you@youraddress.com',
        Subject  => $nothing_to_copy ? "Nothing to Copy" : @INFO ? "Errors in copying" : "Copying completed OK",
        Data     => $nothing_to_copy ? "Nothing to Copy" : @INFO ? join("", @INFO) : "No errors were detected",
);

$msg->send;

Open in new window

0
 
new_perl_userAuthor Commented:
Thanks a ton for all the help.  And finally Two questions:)

1) Is there a way to assign more points than 500  for this question.
2) In the script we are grepping .txt files, Is there a way we can grep file by date because after posting the above requirement I came to know that the filename they supply to me would be in the format.

Everday they are going to give me a new file in the below format and also the old ones  will still be there
 So I need to grep only today's file from that bunch of old files.. All files will be in the below format..

"test_20111003_HDF.txt"
0
 
parparovCommented:
Regarding 500 points, you'll have to ask the admin
We can grep by date, but that would needs to be a carefully formulated separate question.
For instance, whether we should look into real file's date, or the one contained in its name.
0
 
new_perl_userAuthor Commented:
ok. I will ask the admin regarding the points and I will open another question regarding the date of the file.
0
 
new_perl_userAuthor Commented:
Hi,

Opened up a new question "Grep a file by Date". Can you pls let me know the workaround for it..

Thanks,
0
 
parparovCommented:
hope I answered it.
0
 
new_perl_userAuthor Commented:
Hi,

Yes, it is working for grepping a file by date and I accepted it.

But for the above script , is it possible to generate a log file  for the list it is copying  and send as an attachment  in email



0
 
parparovCommented:
Yes, but I think it deserves a separate question. :)
0
 
new_perl_userAuthor Commented:
Thank you and I will open it uo. Spoke with the admin to check if i can assign more points but unfortunately they said no. I am sorry.
0

Featured Post

2018 Annual Membership Survey

Here at Experts Exchange, we strive to give members the best experience. Help us improve the site by taking this survey today! (Bonus: Be entered to win a great tech prize for participating!)

  • 12
  • 11
Tackle projects and never again get stuck behind a technical roadblock.
Join Now