Why does cat >> fail?

Posted on 2004-11-10
Last Modified: 2013-12-26
Hi experts,
I am running into problems with a cat >> command:

$node->runCmd("cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
It fails giving the following error:

cat: /scratch/user/jallmer/slot_2/b.htm: No such file or directory
Failed with status 1 running 'bpsh -n 93 cat /scratch/user/jallmer/slot_2/b.htm >> /home/jallmer/Distrib3/master/mainresult/b.htm'
Failed running 'distribjob /home/jallmer/Distrib3/controller_liniac.prop 93 94 9 90 91' with status '1' at /var/spool/clubmask/1110120250.359920.program line 16.

It probably occurs because the file doesn't exist in the location.
The same cat>>s go on to z.htm.
a.htm works fine ;)
But I cannot garantee that all the files are actually present.
So how can I teach the cat >> command to ignore not existing
files and maybe output an empty file.

Also if there is another command then cat >> which would achieve the result
that would be fine for me, too.
Question by:allmer
    LVL 51

    Assisted Solution

    unset noclobber
    LVL 5

    Author Comment

    I am unfamiliar with UNIX and PERL so please
    make it a little more clear for me.
    LVL 48

    Accepted Solution

    What module is $node->runCmd from?

    You could do:

    if (-f "$nodeExecDir/b.htm") {
      $node->runCmd("cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
    else {
      $node->runCmd("touch $mainResultDir/b.htm");
    LVL 5

    Author Comment

    No idea what module it is, probably an in-house dev.

    Anyway, I added if(-f "..")
    in front of all the cat>>s
    and I just started a test run.
    I'll know the result tomorrow.
    LVL 51

    Expert Comment

    the appplication (most likely a shell script) which starts your perl script, should set noclobber first
    check your man-pages for your shell
    LVL 14

    Expert Comment

    I don't think this is a noclobber issue because a) that would give a different error message and b) noclobber is on csh and Perl is probably calling sh to do this.

    Instead, I'm guessing that the directory doesn't exist.


    $node->runCmd("mkdir -p $mainResultDir; cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");
    LVL 51

    Expert Comment

    bash has noclobber, and linux's sh for example is a softlink to bash :-(
    but I agree with the directory thing
    LVL 5

    Author Comment

    Hi everyone,

    This didn't work:
    if (-f "$nodeExecDir/b.htm") {
      $node->runCmd("cat $nodeExecDir/b.htm >> $mainResultDir/b.htm");

    Same error message.
    I am quite sure that $mainResultDir exists prior to cat >>.
    But I will try the approach.

    I am still not sure about the noclobber.
    Can I execute it like this:
    $node->runCmd("no clobber");?
    I can try that.
    LVL 5

    Author Comment

    If I use
    $node->runCmd("unset noclobber");
    I get an error.
    So this doesn't work.

    If I create the directory before copying I still get an error:
    $node->runCmd("mkdir -p $mainResultDir");

    mkdir: cannot create directory `/home/jallmer/Distrib2/master/mainresult': File exists
    cp: cannot stat `/scratch/user/jallmer/slot_1/*html': No such file or directory
    Failed with status 1 running 'bpsh -n 24 copyRes  /scratch/user/jallmer/slot_1 /home/jallmer/Distrib2/master/mainresult'
    Failed running 'distribjob /home/jallmer/Distrib2/controller_liniac.prop 24 25 26 27 28' with status '1' at /var/spool/clubmask/1111
    162710.121102.program line 16.

    I am pretty sure that the problem is that the files I try to copy or cat are not available in the location I want to copy from.
    So I need some way to either check if they are there or use a copy/ cat method that doesn't care if the files to be copied or concatenated exist or not.

    Any ideas?
    LVL 5

    Author Comment

    Something like:
    if(directory && file exists) {
    } else {
       forget it;
    LVL 48

    Expert Comment

    Why are you now getting the error:

    cp: cannot stat `/scratch/user/jallmer/slot_1/*html': No such file or directory

    Have you changed your copy command?

    How big is this Perl script?  It appears like it's just a glorified wrapper around a shell script?

    If it's not too big, could you please post the code.
    LVL 51

    Expert Comment

    >  If I use
    > $node->runCmd("unset noclobber");
    > I get an error.
    > So this doesn't work.


    > Can I execute it like this:

    Please reread my comment http:#12552601
    You have to set noclobber in your shell script not within perl,
    but see other comments too: you first need to identify your shell where your perl script is called
    LVL 5

    Author Comment

    @ ahoffmann: {
    I am not sure where to unset; see below.
    Not all scripts are controllable by me.
    I can only control the ones displayed below.

    I have a shell script,
    which runs a perl .pl script which in turn runs a shell script
    which then runs a perl .pm module which controls an executable.
    After completion of execution on each node (processor (256) of the server)
    The perl script is supposed to copy my data back.
    That's the problem.
    Here are all the codes:

    First shell script:
    initRun (NumOfProcessorsRequested NameOfQueriesInputFile NameOfDatabase PathToWorkingDirectory)
    mkdir $4
    cd $4
    rm -r $4/*.stdout $4/master/
    /home/jallmer/perldev/DJob/DistribJobTasks/ $1 $2 $3 $4
    The master directory will be created by another perl module.

    Perl script called from above:

    use CBIL::Util::Utils;
    use DJob::DistribJob::Task;
    use strict;
    use File::Basename;
    use POSIX;

    ##set some paths
    my $inputPath = "/home/jallmer/input";

    ##See what's on the cmdline and put into variables.
    die "Usage: $0 maxProcessorNum queryFileName databaseFileName taskDir\n" unless $#ARGV == 3;
    ##Example: 20 queries.qgp chlre2.fasta /home/jallmer/DistribJob
    my ($maxProcessors,$queriesFileName,$dataBaseFileName,$taskDirectory) = @ARGV;

    ##Do necessary calculations
    my $nodes = ceil($maxProcessors/2);
    my $basePath = $taskDirectory;#dirname($taskDirectory);                 #Should point to dir where everything is stored.
    print("Basepath: $basePath\n");

    my $numQueries = ReadInput("$inputPath/$queriesFileName");
    print("Queries: $numQueries\n");
    my $minutes = ceil(($numQueries/$maxProcessors)*8);             #Adjust multiplier when real number is known.
    my $minTaskSize = 20;    #Runs very quickly, probably done in 10-100 minutes.
    my $maxTaskSize = 200;   #Runs for quite a while but probably done over night.
    my $taskSize = ceil($numQueries/$maxProcessors);

    ##Adjust the tasksize according to the number of processors
    ##desired and the number of actual queries to be processed.
    if($taskSize > $maxTaskSize) {
            $taskSize = $maxTaskSize;
    if($taskSize < $minTaskSize) {
            $taskSize = $minTaskSize;
            $maxProcessors = ceil($numQueries/$taskSize);
            $nodes = ceil($maxProcessors/2);
    print("Number of queries: $numQueries\nTaskSize: $taskSize\nNodes: $nodes\n");

    ##Now that the task size is known let's write the files:

    ##output the task.prop file:
    open(F,">$inputPath/task.prop") or die("could not create task.prop");
    print F "dbFilePath=$inputPath/$dataBaseFileName\n";
    print F "queryFilePath=$inputPath/$queriesFileName\n";
    close F;
    ##file output done

    ##output the controller_liniac file
    my $controller = "$basePath/controller_liniac.prop";
    open(F,">$controller") or die("Could not create \"$controller\"");
    print F "masterdir=$basePath/master\n";
    print F "inputdir=$inputPath\n";
    print F 'nodedir=/scratch/user/$ENV{USER}';
    print F "\nslotspernode=2\n";
    print F "subtasksize=$taskSize\n";
    print F "taskclass=DJob::DistribJobTasks::RunGPF\n";
    print F "nodeclass=DJob::DistribJob::BprocNode\n";
    print F "restart=no\n";
    close F;
    ##file output done
    ##Now start the process:
    print("I am starting now, doing this:\nlsubmit $nodes $minutes $taskDirectory\n\n");
    runCmd("liniacsubmit $nodes $minutes $controller");
    #The above is just a bash shortcut see below.
    #alias subTask='liniacsubmit 10 20 /home/jallmer/DistribJob/controller_liniac.prop'
    #this calls the script controlling the server.
    #The .prop file contains the location of my custom perl module (, see below.
    ##should be running, now.

    ##This function simply returns the number of queries from
    ##the input queries file.
    sub ReadInput {
      my($in_file) = @_;
      open(FILE,"$in_file") or die "Couldn't open File \n";
      local $/;
      my @file = split /\r\n|\n\r|\r|\n/,<FILE>;
      my $top;
      my $a = 0;
      while(!($file[$a] =~ m/<QUERIESNUM>/)){
      my $num = $file[$a++];
      $num =~ s/\s+//g;

    #Only the one interesting function here:
    #Namely the one suppopsed to return the results from each node:
    sub integrateSubTaskResults {
      my ($self, $subTaskNum, $node, $nodeExecDir, $mainResultDir) = @_;
      #Copy all info into as many separate folders as processes were running.
      $node->runCmd("copyAll  $nodeExecDir $mainResultDir");
      #Copy the html links into one file
    #  $node->runCmd("cat $nodeExecDir/a.htm >> $mainResultDir/a.html");
    #  ...
    #  $node->runCmd("cat $nodeExecDir/z.htm >> $mainResultDir/z.html");
    # Above apparently doesn't work when no file in the $nodeExecDir.

    #Below never fails but also never returns any results.
    #I am positve that there are results generated by my queries.
    #I can test this not letting it run on the server, but locally.
      if(-f "$nodeExecDir/a.htm") {
        $node->runCmd("cat $nodeExecDir/a.htm >> $mainResultDir/a.html");
    # ...
      if (-f "$nodeExecDir/z.htm") {
        $node->runCmd("cat $nodeExecDir/z.htm >> $mainResultDir/z.html");
      if (-f "$mainResultDir/overview.html") {
       system("cat $mainResultDir/overview.html >> $mainResultDir/00_overview.html");
      if (-f "$mainResultDir/node.fasta") {
        system("cat $mainResultDir/node.fasta >> $mainResultDir/result.fasta");
      #$node->runCmd("cat $nodeExecDir/overview.html >> $mainResultDir/00_overview.html");
      #Copy all predictions into one fasta file.
      #$node->runCmd("cat $nodeExecDir/node.fasta >> $mainResultDir/result.fasta");
      #$node->runCmd("copyAll $nodeExecDir $mainResultDir/output");

    Shell script:
    mkdir $2
    cp $1/* $2
    Maybe this helps clarification.
    Thank you
    LVL 51

    Expert Comment

    mkdir $2

    this does not proof if mkdir fails, that might be a reason why your following copy fails

    BTW, I don't see a good reason why these 2 commands (mkdir and cp) are done in an external shell when they could be done within perl
    LVL 5

    Author Comment

    You are right ahoffmann,
    I don't see a reason for that, either.
    The problem is any wildcard use seems
    to fail with the system I am working on. So I just put it into a shell script.
    LVL 5

    Author Comment

    I guess I got confused with all the different locations, scripts and languages.
    I got it to work, now.
    Tintin's suggestion worked after all.
    Thanks alot everyone for the help!

    Featured Post

    Highfive Gives IT Their Time Back

    Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

    Join & Write a Comment

    Introduction: The undo support, implementing a stack. Continuing from the eigth article about sudoku.   We need a mechanism to keep track of the digits entered so as to implement an undo mechanism.  This should be a ‘Last In First Out’ collec…
    Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
    This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
    It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages.

    746 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    16 Experts available now in Live!

    Get 1:1 Help Now