Why does cat >> fail?

Hi experts,
I am running into problems with a cat >> command:

$node->runCmd("cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
It fails giving the following error:

cat: /scratch/user/jallmer/slot_2/b.htm: No such file or directory
Failed with status 1 running 'bpsh -n 93 cat /scratch/user/jallmer/slot_2/b.htm >> /home/jallmer/Distrib3/master/mainresult/b.htm'
Failed running 'distribjob /home/jallmer/Distrib3/controller_liniac.prop 93 94 9 90 91' with status '1' at /var/spool/clubmask/1110120250.359920.program line 16.

It probably occurs because the file doesn't exist in the location.
The same cat>>s go on to z.htm.
a.htm works fine ;)
But I cannot garantee that all the files are actually present.
So how can I teach the cat >> command to ignore not existing
files and maybe output an empty file.

Also if there is another command then cat >> which would achieve the result
that would be fine for me, too.
Thanks,
Jens
LVL 5
allmerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ahoffmannCommented:
unset noclobber
0
allmerAuthor Commented:
Hi,
I am unfamiliar with UNIX and PERL so please
make it a little more clear for me.
Cheers,
Jens
0
TintinCommented:
What module is $node->runCmd from?

You could do:

if (-f "$nodeExecDir/b.htm") {
  $node->runCmd("cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
}
else {
  $node->runCmd("touch $mainResultDir/b.htm");
}
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

allmerAuthor Commented:
No idea what module it is, probably an in-house dev.

Anyway, I added if(-f "..")
in front of all the cat>>s
and I just started a test run.
I'll know the result tomorrow.
0
ahoffmannCommented:
the appplication (most likely a shell script) which starts your perl script, should set noclobber first
check your man-pages for your shell
0
chris_calabreseCommented:
I don't think this is a noclobber issue because a) that would give a different error message and b) noclobber is on csh and Perl is probably calling sh to do this.

Instead, I'm guessing that the directory doesn't exist.

Try

$node->runCmd("mkdir -p $mainResultDir; cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
0
ahoffmannCommented:
bash has noclobber, and linux's sh for example is a softlink to bash :-(
but I agree with the directory thing
0
allmerAuthor Commented:
Hi everyone,

This didn't work:
if (-f "$nodeExecDir/b.htm") {
  $node->runCmd("cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
}

Same error message.
I am quite sure that $mainResultDir exists prior to cat >>.
But I will try the approach.

I am still not sure about the noclobber.
Can I execute it like this:
$node->runCmd("no clobber");?
I can try that.
0
allmerAuthor Commented:
If I use
$node->runCmd("unset noclobber");
I get an error.
So this doesn't work.

If I create the directory before copying I still get an error:
$node->runCmd("mkdir -p $mainResultDir");

mkdir: cannot create directory `/home/jallmer/Distrib2/master/mainresult': File exists
cp: cannot stat `/scratch/user/jallmer/slot_1/*html': No such file or directory
Failed with status 1 running 'bpsh -n 24 copyRes  /scratch/user/jallmer/slot_1 /home/jallmer/Distrib2/master/mainresult'
Failed running 'distribjob /home/jallmer/Distrib2/controller_liniac.prop 24 25 26 27 28' with status '1' at /var/spool/clubmask/1111
162710.121102.program line 16.

I am pretty sure that the problem is that the files I try to copy or cat are not available in the location I want to copy from.
So I need some way to either check if they are there or use a copy/ cat method that doesn't care if the files to be copied or concatenated exist or not.

Any ideas?
0
allmerAuthor Commented:
Something like:
if(directory && file exists) {
   $node->...;
} else {
   forget it;
}
0
TintinCommented:
Why are you now getting the error:

cp: cannot stat `/scratch/user/jallmer/slot_1/*html': No such file or directory

Have you changed your copy command?

How big is this Perl script?  It appears like it's just a glorified wrapper around a shell script?

If it's not too big, could you please post the code.
0
ahoffmannCommented:
>  If I use
> $node->runCmd("unset noclobber");
> I get an error.
> So this doesn't work.

and

> Can I execute it like this:
NO.

Please reread my comment http:#12552601
You have to set noclobber in your shell script not within perl,
but see other comments too: you first need to identify your shell where your perl script is called
0
allmerAuthor Commented:
@ ahoffmann: {
I am not sure where to unset; see below.
Not all scripts are controllable by me.
I can only control the ones displayed below.
}

I have a shell script,
which runs a perl .pl script which in turn runs a shell script
which then runs a perl .pm module which controls an executable.
After completion of execution on each node (processor (256) of the server)
The perl script is supposed to copy my data back.
That's the problem.
Here are all the codes:

First shell script:
initRun (NumOfProcessorsRequested NameOfQueriesInputFile NameOfDatabase PathToWorkingDirectory)
///////////////////
mkdir $4
cd $4
rm -r $4/*.stdout $4/master/
/home/jallmer/perldev/DJob/DistribJobTasks/initRun.pl $1 $2 $3 $4
//////////////////////
The master directory will be created by another perl module.

Perl script called from above:
initRun.pl
//////////////////////////initRun.pl///////////////////////////
#!/usr/bin/perl

use CBIL::Util::Utils;
use DJob::DistribJob::Task;
use strict;
use File::Basename;
use POSIX;

##set some paths
my $inputPath = "/home/jallmer/input";


##See what's on the cmdline and put into variables.
die "Usage: $0 maxProcessorNum queryFileName databaseFileName taskDir\n" unless $#ARGV == 3;
##Example: 20 queries.qgp chlre2.fasta /home/jallmer/DistribJob
my ($maxProcessors,$queriesFileName,$dataBaseFileName,$taskDirectory) = @ARGV;

##Do necessary calculations
my $nodes = ceil($maxProcessors/2);
my $basePath = $taskDirectory;#dirname($taskDirectory);                 #Should point to dir where everything is stored.
print("Basepath: $basePath\n");

my $numQueries = ReadInput("$inputPath/$queriesFileName");
print("Queries: $numQueries\n");
my $minutes = ceil(($numQueries/$maxProcessors)*8);             #Adjust multiplier when real number is known.
my $minTaskSize = 20;    #Runs very quickly, probably done in 10-100 minutes.
my $maxTaskSize = 200;   #Runs for quite a while but probably done over night.
my $taskSize = ceil($numQueries/$maxProcessors);


##Adjust the tasksize according to the number of processors
##desired and the number of actual queries to be processed.
if($taskSize > $maxTaskSize) {
        $taskSize = $maxTaskSize;
}
if($taskSize < $minTaskSize) {
        $taskSize = $minTaskSize;
        $maxProcessors = ceil($numQueries/$taskSize);
        $nodes = ceil($maxProcessors/2);
}
print("Number of queries: $numQueries\nTaskSize: $taskSize\nNodes: $nodes\n");

##Now that the task size is known let's write the files:

##output the task.prop file:
open(F,">$inputPath/task.prop") or die("could not create task.prop");
print F "dbFilePath=$inputPath/$dataBaseFileName\n";
print F "queryFilePath=$inputPath/$queriesFileName\n";
close F;
##file output done

##output the controller_liniac file
my $controller = "$basePath/controller_liniac.prop";
open(F,">$controller") or die("Could not create \"$controller\"");
print F "masterdir=$basePath/master\n";
print F "inputdir=$inputPath\n";
print F 'nodedir=/scratch/user/$ENV{USER}';
print F "\nslotspernode=2\n";
print F "subtasksize=$taskSize\n";
print F "taskclass=DJob::DistribJobTasks::RunGPF\n";
print F "nodeclass=DJob::DistribJob::BprocNode\n";
print F "restart=no\n";
close F;
##file output done
##Now start the process:
print("I am starting now, doing this:\nlsubmit $nodes $minutes $taskDirectory\n\n");
runCmd("liniacsubmit $nodes $minutes $controller");
#The above is just a bash shortcut see below.
#.bashrc:
#alias subTask='liniacsubmit 10 20 /home/jallmer/DistribJob/controller_liniac.prop'
#this calls the script controlling the server.
#The .prop file contains the location of my custom perl module (RunGPF.pm), see below.
##should be running, now.

##This function simply returns the number of queries from
##the input queries file.
sub ReadInput {
  my($in_file) = @_;
  open(FILE,"$in_file") or die "Couldn't open File \n";
  local $/;
  my @file = split /\r\n|\n\r|\r|\n/,<FILE>;
  my $top;
  my $a = 0;
  while(!($file[$a] =~ m/<QUERIESNUM>/)){
    $top.=$file[$a++];
  }
  $top.=$file[$a++];
  my $num = $file[$a++];
  $num =~ s/\s+//g;
  close(FILE);
  return($num);
}
/////////////////////////initRun.pl////////////////////////////////

//////////////////////RunGPF.pm////////////////////////////////
#Only the one interesting function here:
#Namely the one suppopsed to return the results from each node:
sub integrateSubTaskResults {
  my ($self, $subTaskNum, $node, $nodeExecDir, $mainResultDir) = @_;
  #Copy all info into as many separate folders as processes were running.
  $node->runCmd("copyAll  $nodeExecDir $mainResultDir");
  #Copy the html links into one file
#  $node->runCmd("cat $nodeExecDir/a.htm >> $mainResultDir/a.html");
#  ...
#  $node->runCmd("cat $nodeExecDir/z.htm >> $mainResultDir/z.html");
# Above apparently doesn't work when no file in the $nodeExecDir.

#Below never fails but also never returns any results.
#I am positve that there are results generated by my queries.
#I can test this not letting it run on the server, but locally.
  if(-f "$nodeExecDir/a.htm") {
    $node->runCmd("cat $nodeExecDir/a.htm >> $mainResultDir/a.html");
  }
# ...
  if (-f "$nodeExecDir/z.htm") {
    $node->runCmd("cat $nodeExecDir/z.htm >> $mainResultDir/z.html");
  }
  if (-f "$mainResultDir/overview.html") {
   system("cat $mainResultDir/overview.html >> $mainResultDir/00_overview.html");
  }
  if (-f "$mainResultDir/node.fasta") {
    system("cat $mainResultDir/node.fasta >> $mainResultDir/result.fasta");
  }
  #$node->runCmd("cat $nodeExecDir/overview.html >> $mainResultDir/00_overview.html");
  #Copy all predictions into one fasta file.
  #$node->runCmd("cat $nodeExecDir/node.fasta >> $mainResultDir/result.fasta");
  #$node->runCmd("copyAll $nodeExecDir $mainResultDir/output");
}
1;
/////////////////////RunGPF.pm////////////////////////////////

Shell script:
////////////////////CopyAll////////////////////////////////////
#!/bin/sh
mkdir $2
cp $1/* $2
////////////////////CopyAll////////////////////////////////////
Maybe this helps clarification.
Thank you
Jens
0
ahoffmannCommented:
////////////////////CopyAll////////////////////////////////////
mkdir $2

this does not proof if mkdir fails, that might be a reason why your following copy fails

BTW, I don't see a good reason why these 2 commands (mkdir and cp) are done in an external shell when they could be done within perl
0
allmerAuthor Commented:
You are right ahoffmann,
I don't see a reason for that, either.
The problem is any wildcard use seems
to fail with the system I am working on. So I just put it into a shell script.
0
allmerAuthor Commented:
Anyway,
I guess I got confused with all the different locations, scripts and languages.
I got it to work, now.
Tintin's suggestion worked after all.
Thanks alot everyone for the help!
Jens
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
System Programming

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.