Extracting certian lines from a fixed length text file

Posted on 2003-10-28
Last Modified: 2010-04-22
I have a fixed length text file I need to "Prune".  There is a date stamp in each record.  Characters 141-144 are a 4 digit year.  Chars 145-146 are two digit month and 147-148 are two digit day.

I would like to use "sed" or "awk" to create a new file FROM this file for all records with a date greater than or equal to 01/01/2003 (for example)

I've looked at both "sed" and "awk" and sort of understand how they work, but can't quite figure out how to get the fields from my file when I don't really have any field delimiter.

My ultimate goal is to have a shell script that will accept 3 parmaters.  "date", "inFile" and "outFile"

Question by:andersen58
LVL 38

Expert Comment

ID: 9638530
It is a lot easy to do it with "cut" if you want to use "date" as paramter and ckeck it
agains the records in file.

Here's the example script

# useage: $0 date(dd/mm/yyyy) Infile OutFile"


DD=`echo "$1 | cut -f1 -d/"`
MM=`echo "$1 | cut -f2 -d/"`
YY=`echo "$1 | cut -f2 -d/"`

#make sure OUFILE is blank, if you want to append records to this file
#just comment it out

cat /dev/null > $OUFILE

#Now read the input file, and  get the records
IFS="\012"    #read in the whole line, just in case you have white space in the records

exec 0<$INFILE

while read RECORD ; do
         RDD=`echo $RECORD | cut -c147-148 `
         RMM=`echo $RECORD | cut -c145-146 `
         RYY=`echo $RECORD | cut -c141-144 `
if [ "$RYY" -gt "$YY" ] ; then
         echo "$RECORD" >> $OUFILE         # > 2003
              if [ "$RYY" -eq "$YY" -a "$RMM" -gt "$MM" ] ; then
                     echo "$RECORD" >> $OUFILE         #
                   if [ "$RMM" -eq "$MM" -a  "$RDD" -ge "$DD" ] ; then
                         echo "$RECORD" >> $OUFILE        

# End of script

 you can also, modify the script to use:

if command1 ; then
elif command2 ; then
elif commandn ; then

LVL 23

Expert Comment

ID: 9638536

Accepted Solution

glassd earned 500 total points
ID: 9639988
You can use awk. It depends on your test date format. Assuming the format is as given (DD/MM/YYYY), and is held in the shell variable $Date:

awk -v d=$Date  '
      if(b <= t) {
         print $0
  }' infile > outfile

The BEGIN section takes the variable d, which is passed into awk, and splits it into the array a[] using the / as a separator. The three elements are then reassembled in the correct order using the sprinf() function. Note that I have assumed UK date format as opposed to US format, so you may have to change this if you have MM/DD/YYYY.

The main section compares this date with the 8 character date string extracted from each line using the substr() function. If the test succeeds then the whole line is printed.


Expert Comment

ID: 9640010
Oops, sorry, got my test the wrong way round.

       if(b >= t) {

Author Comment

ID: 9645828
Thanks glassd

  Added just a few lines to pass in command line variables and it worked like great.  Ran through a 5+ MB file and created a 1.6 MB "pruned down" file in about 1 second!

It don't get any better than that!

Good work.

PS. Yeah, I caught the test being backwards just AFTER I run it (ha!)
Looked at the output and went "Hmm, that ain't right"... Easy fix!

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
benchmark write speed to disk on linux 2 66
How to use docker to run your java program on a linux box 9 99
Error while installing rpm 1 77
Use Powershell script to ftp 10 83
Have you ever been frustrated by having to click seven times in order to retrieve a small bit of information from the web, always the same seven clicks, scrolling down and down until you reach your target? When you know the benefits of the command l…
The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
Internet Business Fax to Email Made Easy - With eFax Corporate (, you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
Internet Business Fax to Email Made Easy - With  eFax Corporate (, you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now