Nawk outputing the error: "nawk: FILENAME makes too many open files"

Hi All,

For some background this question has the following PAQ's leading up to this point.
This question isn't so much about how to refine the awk script, rather than to determine how best to work with large files.

The file I am testing gets broken into 30+ files, but breaks at the 20 mark. If you're interested in the input in order to answer the question, please review the following PAQ's.

http://www.experts-exchange.com/Programming/Programming_Platforms/Unix_Programming/Q_21763311.html
http://www.experts-exchange.com/Programming/Programming_Platforms/Unix_Programming/Q_21764891.html
And most recently:
http://www.experts-exchange.com/Programming/Programming_Platforms/Unix_Programming/Q_21854851.html

----------------------------------------------------------------------

SCRIPT EXPLANATION:
===============

The current nawk is formatted:

=================================================
nawk -F| -v Footer= BEGIN{f="/dev/null"; oldfile = f}
                /^\|\|/{
                        c++
                        f=sprintf("%s%s'$DateTime'%03d.DHR",$3,$4,c)
                        print Footer >> oldfile
                        oldfile = f
                       }

                       { print >> f }
                END { print Footer >> oldfile }
                FILENAME
=================================================

Where the files being created look like:

AU402679060529.1643001.DHR
AU414381060529.1643002.DHR
AU420252060529.1643003.DHR
AU423247060529.1643004.DHR
<etc....>

Or general format:
XX123456YYMMDD.HHMM???.DHR
Where:
XX => Comes from the file delimiter
123456 => Comes from the delimiter
YYMMDD.HHMM => Comes from a variable $DateTime

In addition:
* the variable $Footer contains some text to be inserted at the end of each file.
* The delimiter is || where example is ||XX|123456|blah|blah|blah|blah
* /dev/null is used to ensure all text in input file before the first "||" is ignored

ERROR MESSAGE AS IT APPEAR:
=====================

This currently appears upon execution of a large input file (from ksh -x output):

=================================================
+ nawk -F| -v Footer= BEGIN{f="/dev/null"; oldfile = f}
                /^\|\|/{
                        c++
                        f=sprintf("%s%s060529.1639%03d.DHR",$3,$4,c)
                        print Footer >> oldfile
                        oldfile = f
                       }

                       {print >> f }
                END { print Footer >> oldfile}

                /directory/filename
nawk: AU451922060529.1639020.DHR makes too many open files
 input record number 7676, file /directory/filename
 source line number 9
=================================================

QUESTIONS:

Q: How can this error be avoided?
Q: Is there a more efficient way to deal with large files?


Thanks in advance,

GlennStart Free Trial