Why does cat >> fail?

Posted on 2004-11-10
Medium Priority
Last Modified: 2013-12-26
Hi experts,
I am running into problems with a cat >> command:

$node->runCmd("cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
It fails giving the following error:

cat: /scratch/user/jallmer/slot_2/b.htm: No such file or directory
Failed with status 1 running 'bpsh -n 93 cat /scratch/user/jallmer/slot_2/b.htm >> /home/jallmer/Distrib3/master/mainresult/b.htm'
Failed running 'distribjob /home/jallmer/Distrib3/controller_liniac.prop 93 94 9 90 91' with status '1' at /var/spool/clubmask/1110120250.359920.program line 16.

It probably occurs because the file doesn't exist in the location.
The same cat>>s go on to z.htm.
a.htm works fine ;)
But I cannot garantee that all the files are actually present.
So how can I teach the cat >> command to ignore not existing
files and maybe output an empty file.

Also if there is another command then cat >> which would achieve the result
that would be fine for me, too.
Question by:allmer
  • 8
  • 5
  • 2
  • +1
LVL 51

Assisted Solution

ahoffmann earned 200 total points
ID: 12548571
unset noclobber

Author Comment

ID: 12548835
I am unfamiliar with UNIX and PERL so please
make it a little more clear for me.
LVL 48

Accepted Solution

Tintin earned 800 total points
ID: 12549401
What module is $node->runCmd from?

You could do:

if (-f "$nodeExecDir/b.htm") {
  $node->runCmd("cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
else {
  $node->runCmd("touch $mainResultDir/b.htm");
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.


Author Comment

ID: 12549715
No idea what module it is, probably an in-house dev.

Anyway, I added if(-f "..")
in front of all the cat>>s
and I just started a test run.
I'll know the result tomorrow.
LVL 51

Expert Comment

ID: 12552601
the appplication (most likely a shell script) which starts your perl script, should set noclobber first
check your man-pages for your shell
LVL 14

Expert Comment

ID: 12555645
I don't think this is a noclobber issue because a) that would give a different error message and b) noclobber is on csh and Perl is probably calling sh to do this.

Instead, I'm guessing that the directory doesn't exist.


$node->runCmd("mkdir -p $mainResultDir; cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
LVL 51

Expert Comment

ID: 12557670
bash has noclobber, and linux's sh for example is a softlink to bash :-(
but I agree with the directory thing

Author Comment

ID: 12558417
Hi everyone,

This didn't work:
if (-f "$nodeExecDir/b.htm") {
  $node->runCmd("cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");

Same error message.
I am quite sure that $mainResultDir exists prior to cat >>.
But I will try the approach.

I am still not sure about the noclobber.
Can I execute it like this:
$node->runCmd("no clobber");?
I can try that.

Author Comment

ID: 12560242
If I use
$node->runCmd("unset noclobber");
I get an error.
So this doesn't work.

If I create the directory before copying I still get an error:
$node->runCmd("mkdir -p $mainResultDir");

mkdir: cannot create directory `/home/jallmer/Distrib2/master/mainresult': File exists
cp: cannot stat `/scratch/user/jallmer/slot_1/*html': No such file or directory
Failed with status 1 running 'bpsh -n 24 copyRes  /scratch/user/jallmer/slot_1 /home/jallmer/Distrib2/master/mainresult'
Failed running 'distribjob /home/jallmer/Distrib2/controller_liniac.prop 24 25 26 27 28' with status '1' at /var/spool/clubmask/1111
162710.121102.program line 16.

I am pretty sure that the problem is that the files I try to copy or cat are not available in the location I want to copy from.
So I need some way to either check if they are there or use a copy/ cat method that doesn't care if the files to be copied or concatenated exist or not.

Any ideas?

Author Comment

ID: 12561115
Something like:
if(directory && file exists) {
} else {
   forget it;
LVL 48

Expert Comment

ID: 12561883
Why are you now getting the error:

cp: cannot stat `/scratch/user/jallmer/slot_1/*html': No such file or directory

Have you changed your copy command?

How big is this Perl script?  It appears like it's just a glorified wrapper around a shell script?

If it's not too big, could you please post the code.
LVL 51

Expert Comment

ID: 12563163
>  If I use
> $node->runCmd("unset noclobber");
> I get an error.
> So this doesn't work.


> Can I execute it like this:

Please reread my comment http:#12552601
You have to set noclobber in your shell script not within perl,
but see other comments too: you first need to identify your shell where your perl script is called

Author Comment

ID: 12568631
@ ahoffmann: {
I am not sure where to unset; see below.
Not all scripts are controllable by me.
I can only control the ones displayed below.

I have a shell script,
which runs a perl .pl script which in turn runs a shell script
which then runs a perl .pm module which controls an executable.
After completion of execution on each node (processor (256) of the server)
The perl script is supposed to copy my data back.
That's the problem.
Here are all the codes:

First shell script:
initRun (NumOfProcessorsRequested NameOfQueriesInputFile NameOfDatabase PathToWorkingDirectory)
mkdir $4
cd $4
rm -r $4/*.stdout $4/master/
/home/jallmer/perldev/DJob/DistribJobTasks/initRun.pl $1 $2 $3 $4
The master directory will be created by another perl module.

Perl script called from above:

use CBIL::Util::Utils;
use DJob::DistribJob::Task;
use strict;
use File::Basename;
use POSIX;

##set some paths
my $inputPath = "/home/jallmer/input";

##See what's on the cmdline and put into variables.
die "Usage: $0 maxProcessorNum queryFileName databaseFileName taskDir\n" unless $#ARGV == 3;
##Example: 20 queries.qgp chlre2.fasta /home/jallmer/DistribJob
my ($maxProcessors,$queriesFileName,$dataBaseFileName,$taskDirectory) = @ARGV;

##Do necessary calculations
my $nodes = ceil($maxProcessors/2);
my $basePath = $taskDirectory;#dirname($taskDirectory);                 #Should point to dir where everything is stored.
print("Basepath: $basePath\n");

my $numQueries = ReadInput("$inputPath/$queriesFileName");
print("Queries: $numQueries\n");
my $minutes = ceil(($numQueries/$maxProcessors)*8);             #Adjust multiplier when real number is known.
my $minTaskSize = 20;    #Runs very quickly, probably done in 10-100 minutes.
my $maxTaskSize = 200;   #Runs for quite a while but probably done over night.
my $taskSize = ceil($numQueries/$maxProcessors);

##Adjust the tasksize according to the number of processors
##desired and the number of actual queries to be processed.
if($taskSize > $maxTaskSize) {
        $taskSize = $maxTaskSize;
if($taskSize < $minTaskSize) {
        $taskSize = $minTaskSize;
        $maxProcessors = ceil($numQueries/$taskSize);
        $nodes = ceil($maxProcessors/2);
print("Number of queries: $numQueries\nTaskSize: $taskSize\nNodes: $nodes\n");

##Now that the task size is known let's write the files:

##output the task.prop file:
open(F,">$inputPath/task.prop") or die("could not create task.prop");
print F "dbFilePath=$inputPath/$dataBaseFileName\n";
print F "queryFilePath=$inputPath/$queriesFileName\n";
close F;
##file output done

##output the controller_liniac file
my $controller = "$basePath/controller_liniac.prop";
open(F,">$controller") or die("Could not create \"$controller\"");
print F "masterdir=$basePath/master\n";
print F "inputdir=$inputPath\n";
print F 'nodedir=/scratch/user/$ENV{USER}';
print F "\nslotspernode=2\n";
print F "subtasksize=$taskSize\n";
print F "taskclass=DJob::DistribJobTasks::RunGPF\n";
print F "nodeclass=DJob::DistribJob::BprocNode\n";
print F "restart=no\n";
close F;
##file output done
##Now start the process:
print("I am starting now, doing this:\nlsubmit $nodes $minutes $taskDirectory\n\n");
runCmd("liniacsubmit $nodes $minutes $controller");
#The above is just a bash shortcut see below.
#alias subTask='liniacsubmit 10 20 /home/jallmer/DistribJob/controller_liniac.prop'
#this calls the script controlling the server.
#The .prop file contains the location of my custom perl module (RunGPF.pm), see below.
##should be running, now.

##This function simply returns the number of queries from
##the input queries file.
sub ReadInput {
  my($in_file) = @_;
  open(FILE,"$in_file") or die "Couldn't open File \n";
  local $/;
  my @file = split /\r\n|\n\r|\r|\n/,<FILE>;
  my $top;
  my $a = 0;
  while(!($file[$a] =~ m/<QUERIESNUM>/)){
  my $num = $file[$a++];
  $num =~ s/\s+//g;

#Only the one interesting function here:
#Namely the one suppopsed to return the results from each node:
sub integrateSubTaskResults {
  my ($self, $subTaskNum, $node, $nodeExecDir, $mainResultDir) = @_;
  #Copy all info into as many separate folders as processes were running.
  $node->runCmd("copyAll  $nodeExecDir $mainResultDir");
  #Copy the html links into one file
#  $node->runCmd("cat $nodeExecDir/a.htm >> $mainResultDir/a.html");
#  ...
#  $node->runCmd("cat $nodeExecDir/z.htm >> $mainResultDir/z.html");
# Above apparently doesn't work when no file in the $nodeExecDir.

#Below never fails but also never returns any results.
#I am positve that there are results generated by my queries.
#I can test this not letting it run on the server, but locally.
  if(-f "$nodeExecDir/a.htm") {
    $node->runCmd("cat $nodeExecDir/a.htm >> $mainResultDir/a.html");
# ...
  if (-f "$nodeExecDir/z.htm") {
    $node->runCmd("cat $nodeExecDir/z.htm >> $mainResultDir/z.html");
  if (-f "$mainResultDir/overview.html") {
   system("cat $mainResultDir/overview.html >> $mainResultDir/00_overview.html");
  if (-f "$mainResultDir/node.fasta") {
    system("cat $mainResultDir/node.fasta >> $mainResultDir/result.fasta");
  #$node->runCmd("cat $nodeExecDir/overview.html >> $mainResultDir/00_overview.html");
  #Copy all predictions into one fasta file.
  #$node->runCmd("cat $nodeExecDir/node.fasta >> $mainResultDir/result.fasta");
  #$node->runCmd("copyAll $nodeExecDir $mainResultDir/output");

Shell script:
mkdir $2
cp $1/* $2
Maybe this helps clarification.
Thank you
LVL 51

Expert Comment

ID: 12570704
mkdir $2

this does not proof if mkdir fails, that might be a reason why your following copy fails

BTW, I don't see a good reason why these 2 commands (mkdir and cp) are done in an external shell when they could be done within perl

Author Comment

ID: 12585529
You are right ahoffmann,
I don't see a reason for that, either.
The problem is any wildcard use seems
to fail with the system I am working on. So I just put it into a shell script.

Author Comment

ID: 12585550
I guess I got confused with all the different locations, scripts and languages.
I got it to work, now.
Tintin's suggestion worked after all.
Thanks alot everyone for the help!

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This is to be the first in a series of articles demonstrating the development of a complete windows based application using the MFC classes.  I’ll try to keep each article focused on one (or a couple) of the tasks that one may meet.   Introductio…
Introduction: Database storage, where is the exe actually on the disc? Playing a game selected randomly (how to generate random numbers).  Error trapping with try..catch to help the code run even if something goes wrong. Continuing from the seve…
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
Is your OST file inaccessible, Need to transfer OST file from one computer to another? Want to convert OST file to PST? If the answer to any of the above question is yes, then look no further. With the help of Stellar OST to PST Converter, you can e…
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question