Solved

How can I get a formatted directory listing using awk?

Posted on 2003-11-26
12
2,441 Views
Last Modified: 2013-12-26
I want to get a formatted directory listing that I can parse using other utilities such as perl, Excel, etc.  The unix "ls" command does not provide an easy way to accomplish this.  I am collecting the listings from servers running Linux, Solaris, HP-UX, or AIX.  All four OS's have some form of awk on it.

I would prefer the listing to be delimited and contain the full path names...  i.e.

permissions|owner|group|size|date|/full/path/to/file

example:
-rwxr-xr-x|root|system|125713|Sep 14 12:23|/usr/local/bin/sshd2
-rwxr-xr-x|root|system|125021|Sep 14 12:23|/usr/local/bin/scp2

Can this be done and how?  A working example would be great!  I could do this in perl, but I am not guaranteed that perl is available on all of the systems.  Also, if you can suggest another way of doing this for all 4 OS's if it is not possible with awk, that would be cool too!

Thanks!
0
Comment
Question by:ddurmon
  • 5
  • 3
  • 3
  • +1
12 Comments
 
LVL 9

Expert Comment

by:HamdyHassan
ID: 9825952
Idea,
you can use cut -c command to parse the output of ls -lt command
I think all these OS's has cut command, but the difference could be positions

Another idea, to use Perl

 
0
 
LVL 9

Expert Comment

by:HamdyHassan
ID: 9826077
Using Perl

parse_dir - parse directory listing

SUPPORTED PLATFORMS
Linux
Solaris
Windows


SYNOPSIS
 use File::Listing;
 for (parse_dir(`ls -l`)) {
     ($name, $type, $size, $mtime, $mode) = @$_;
     next if $type ne 'f'; # plain file
     #...
 }
 # directory listing can also be read from a file
 open(LISTING, "zcat ls-lR.gz|");
 $dir = parse_dir(\*LISTING, '+0000');


DESCRIPTION
The parse_dir() routine can be used to parse directory listings. Currently it only understand Unix 'ls -l' and 'ls -lR' format. It should eventually be able to most things you might get back from a ftp server file listing (LIST command), i.e. VMS listings, NT listings, DOS listings,...

The first parameter to parse_dir() is the directory listing to parse. It can be a scalar, a reference to an array of directory lines or a glob representing a filehandle to read the directory listing from.

The second parameter is the time zone to use when parsing time stamps in the listing. If this value is undefined, then the local time zone is assumed.

The third parameter is the type of listing to assume. The values will be strings like 'unix', 'vms', 'dos'. Currently only 'unix' is implemented and this is also the default value. Ideally, the listing type should be determined automatically.

The fourth parameter specifies how unparseable lines should be treated. Values can be 'ignore', 'warn' or a code reference. Warn means that the perl warn() function will be called. If a code reference is passed, then this routine will be called and the return value from it will be incorporated in the listing. The default is 'ignore'.

Only the first parameter is mandatory.

The return value from parse_dir() is a list of directory entries. In a scalar context the return value is a reference to the list. The directory entries are represented by an array consisting of [ $filename, $filetype, $filesize, $filetime, $filemode ]. The $filetype value is one of the letters 'f', 'd', 'l' or '?'. The $filetime value is the seconds since Jan 1, 1970. The $filemode is a bitmask like the mode returned by stat().



0
 
LVL 9

Expert Comment

by:HamdyHassan
ID: 9826247


Example
The following script will list "filename" , "|" , "file size"

#!/export/vol/bin/perl
use File::Listing;
 for (parse_dir(`ls -l`)) {
     ($name, $type, $size, $mtime, $mode) = @$_;
     next if $type ne 'f'; # plain file
     print $name . "|" . $size .  "\n" ;
 }
0
 

Author Comment

by:ddurmon
ID: 9826660
I would love to use perl, but perl is not installed on all of the systems (and I don't control the system so I can't install it).  Is there a way using built-in utilities like awk?
0
 

Author Comment

by:ddurmon
ID: 9826686
P.S.  The reason I don't control the systems is that I am an Internal Auditor (I used to be a System Admin, but I'm not any good with awk beyond the very basic stuff).  I need to collect the listings to support my audit work.
0
 
LVL 7

Expert Comment

by:glassd
ID: 9828010
To get the full path names of a single directory you would have to do something like this:

#!/bin/ksh

ThisDir=$(pwd)

ls -l | awk -v ThisDir=$ThisDir '{
printf("%s|%s|%s|%s|%s %s %s|%s|%s/%s\n",$1,$2,$3,$4,$5,$6,$7,$8,ThisDir,$9)
}'

There are neater ways, but see if this does what you want first.
If you need to do a recursive listing through sub directories then I think this could be done.

I would stick to ksh if you are not sure what shells are likely to be available.

0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 7

Assisted Solution

by:glassd
glassd earned 250 total points
ID: 9828178
A couple of corrections (your required format isn't quite the same as the ls -l output, certainly on Solaris.

Also some awk versions are more capable than others. For example in Solaris you would need to use nawk, and in Linux you may need gawk.

I don't know the uname replies for OS versions other than Solaris, so you may need to do some editting.

#!/bin/ksh

ThisDir=$(pwd)
OS=$(uname -s)

case $OS in
   SunOS)   AWK=nawk;;
   Linux)     AWK=gawk;;
   *)           AWK=awk;;
esac

ls -l | $AWK -v ThisDir=$ThisDir '{
   printf("%s|%s|%s|%s|%s %s %s|%s/%s\n",$1,$3,$4,$8,$5,$6,$7,ThisDir,$9)
}'

This would fail if the file names contained spaces. If that is a problem then there is always a solution.
0
 

Author Comment

by:ddurmon
ID: 9828372
Perfect!  ...  Now how do I get it to do the list recursively?  :-)  
I can do this:


#!/bin/ksh
IFS='
'
RootDir=$1
for dir in $(find $RootDir -type d -print); do
  cd "$dir"
  ThisDir=$(pwd)
  ls -la "$ThisDir" | awk -v ThisDir=$ThisDir '{
  printf("%s|%s|%s|%s|%s|%s %s %s|%s/%s\n",$1,$2,$3,$4,$5,$6,$7,$8,ThisDir,$9)
  }' | grep -v "total|"
done


and it works, but is there a easier/faster/cheaper/better way to do it?
0
 

Author Comment

by:ddurmon
ID: 9828386
I do realize that I am going to have to tweak it a little for each OS.  Linux did this:

-rw-------|1|ddurmon|sysadmin|116 Feb 21|2002|/home/ddurmon/.Xauthority

instead of this:

-rw-------|1|ddurmon|sysadmin|116|Feb 21 2002|/home/ddurmon/.Xauthority

So I had to tweak the printf line...  :-)

0
 
LVL 1

Accepted Solution

by:
mbekker earned 250 total points
ID: 9830485
Hi ddurmon,

For AIX you could try the following, it's more compact and will possibly also work on other systems (except for the date-part, which is different as you found out):

RootDir=$1
find $RootDir -type f -ls 2>/dev/null | awk '{printf "%s|%s|%s|%s|%s|%s %s %s|%s\n",$3,$4,$5,$6,$7,$8,$9,$10,$11}'

If you want it recursively just put " | sort -t'|' +6 " behind it ;-)

Good luck!
0
 
LVL 7

Expert Comment

by:glassd
ID: 9830639
May be you need to include the whole awk statement inside the case construct, rather than just setting the name of the executable there.
0
 

Author Comment

by:ddurmon
ID: 9833822
Thank you glassd and mbekkler.  This is exactly what I was looking for!  :-)

Thanks,

ddurmon
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Introduction: Ownerdraw of the grid button.  A singleton class implentation and usage. Continuing from the fifth article about sudoku.   Open the project in visual studio. Go to the class view – CGridButton should be visible as a class.  R…
Introduction: Dialogs (1) modal - maintaining the database. Continuing from the ninth article about sudoku.   You might have heard of modal and modeless dialogs.  Here with this Sudoku application will we use one of each type: a modal dialog …
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now