Solved

Accepting folder, filename or wildcard for python argument

Posted on 2011-03-11
18
1,460 Views
Last Modified: 2012-05-11
I'm looking to take input from the command in the form of a folder, filename or wildcard entry for the script to use. Help getting it to work and the IF ELSE statement corrected would be appreciated. I have not determined how the directory or wildcard entry will be handled yet.

from optparse import OptionParser

parser = OptionParser(description='Specify directory or files')
parser.add_argument('-d', '', action='store', dest='FOLDER', default=False,
                   help='specify directory'
                   )
parser.add_argument('-f', action='store', dest='LOCAL_PATH',
                   help='specify file'
                   )
parser.add_argument('--version', action='version', version='%(prog)s 1.0')
(options, args) = parser.parse_args(sys.argv)

fname = args[1] # name of folder or file

if options.FOLDER:
FOLDER = LOCAL_PATH:
else:
LOCAL_PATH = '/backups/'

Open in new window

0
Comment
Question by:adamshields
  • 10
  • 8
18 Comments
 
LVL 16

Expert Comment

by:gelonida
ID: 35118257
You are not very clear, what you want to obtain exactly.

Please specify as well, whether your script should run under windows or under linux.

The reason why I a'm asking is, that under linux wildcards will already be expanded by the command line.

Under Linux:
assume the current directory contains the files
a.txt
b.txt
myscript.py

if you call ./myscript.py *.txt
then sys.argsv will be ['./myscript.py', 'a.txt', 'b.txt' ] in Linux
but
['C:\....\myscript.py'm ;*,txt'] under windows

If you want, that the python scrupt receies the wildcards under linux, then you had to call the script with single quotes around the wildcards

./myscript.py '*.txt'




Please tell me exactly in which of the following ways you want to call the script and what you expect
or even better give me some examples.

1.)
script.py *.txt    # shall expand the wildcards and do something for each file in the current directory.

2.)
script.py -d directoryname # shall do something with all files of the specified directory?

3.)
script.py -d dirname *.txt # shall search for the given files in a directory

0
 
LVL 16

Expert Comment

by:gelonida
ID: 35118396
Here a first guess of what you might be looking for.
from optparse import OptionParser
import glob

LOCAL_PATH = '/backups/'

parser = OptionParser(description='Specify directory or files')
parser.add_argument('-d', '--directory', action='store', dest='directory', 
        default=LOCAL_PATH,
        help='specify directory'
)
parser.add_argument('-f', '--file',
    action='store', dest='filename',
    default=None,
    help='specify file'
)
parser.add_argument('--version', action='version', version='%(prog)s 1.0')
(options, args) = parser.parse_args(sys.argv[1:0])

filenames_or_wildcards = args # take all filenames passed in the command line

# This is just an example and seems rather useless, 
# I assume I misunderstood what you wanted 
if options.filename is not None:
    filenames_or_wildcards.append(options.filename)


# Now expand all wildcards
# glob.glob transforms a filename or a wildcard in a list of all matches
# a non existing filename will be 'dropped'
all_files = []
for filename_or_wildcard in filenames_or_wildcards:
    all_files.extend( glob.glob(filename_or_wildcard) )

Open in new window

import os 
import sys
import glob 
from optparse import OptionParser

LOCAL_PATH = '/backups/'

parser = OptionParser()
parser.add_option('-d', '--directory', action='store', dest='directory', 
        default=LOCAL_PATH,
        help='specify directory'
)
parser.add_option('-f', '--file',
    action='store', dest='filename',
    default=None,
    help='specify file'
)
#parser.add_option('--version', action='version', version='%(prog)s 1.0')

options, args = parser.parse_args(sys.argv[1:])

# just guessing.  Might not be what you want
os.chdir( options.directory)

filenames_or_wildcards = args # take all filenames passed in the command line

# This is just an example and seems rather useless, 
# I assume I misunderstood what you wanted 
if options.filename is not None:
    filenames_or_wildcards.append(options.filename)

# Now expand all wildcards
# glob.glob transforms a filename or a wildcard in a list of all matches
# a non existing filename will be 'dropped'
all_files = []
for filename_or_wildcard in filenames_or_wildcards:
    all_files.extend( glob.glob(filename_or_wildcard) )

print "do something with", all_files

Open in new window

0
 
LVL 16

Expert Comment

by:gelonida
ID: 35118402
Please ignore the first code snippet.
The second one is the one to look at
0
 
LVL 3

Author Comment

by:adamshields
ID: 35140128
gelonida, I think it's close. Here's what I have in the script for the LOCAL_PATH functionality that I was using. How should this be modified to use LOCAL_PATH if default action is taken or to use all_files if one the arguments if specified?

# List files in directory and upload them to bucket
for filename in os.listdir(LOCAL_PATH):
        k = Key(bucket)
        k.key = filename
        k.set_contents_from_filename(LOCAL_PATH + filename, cb=percent_cb, num_cb=10)

Open in new window

0
 
LVL 16

Expert Comment

by:gelonida
ID: 35164799
Hmm if you look at my code snippet you see, that the default value of
options.directory is already LOCAL_PATH

so the only thing, that is missing is to add a '*'  wildcard in the case, that
filenames_or_wildcards is empty.

So you could insert following line directly after line 30

if len(filenames_or_wildcards) == 0:
  filenames_or_wildcards = '*'

lines 36/37 would then expand the '*' to all files in LOCAL_DIR


At the end of your snippet you could now do something with all files

for filename in all_files:

        #skip all directory entries which are not a file
        if not os.path.isfile(filename):
              continue
        # alternative could be to skip only directories and to tread
        #  symbolic links and named pipes and devices.
        #if os.path.isdir(filename):
         #     continue

        k = Key(bucket)
        k.key = filename
        k.set_contents_from_filename(LOCAL_PATH + filename, cb=percent_cb, num_cb=10)

I hope I understood correctly what you try to achieve.



from optparse import OptionParser

parser = OptionParser(description='Specify directory or files')
parser.add_argument('-d', action='store', dest='folder', default=False,
                   help='specify directory'
                   )
parser.add_argument('-f', action='store', dest='LOCAL_PATH',
                   help='specify file'
                   )
parser.add_argument('--version', action='version', version='%(prog)s 1.0')
(options, args) = parser.parse_args(sys.argv)

fname = args[1] # name of folder or file

if options.FOLDER:
FOLDER = LOCAL_PATH:
else:
LOCAL_PATH = '/backups/'

Open in new window

0
 
LVL 3

Author Comment

by:adamshields
ID: 35165503
The -d and default are working but the -f where a single file is specified does not. It seems to be a logical error as the script is not throwing an exception. Do you see any issues? Also the --version throws an exception when un-commented, any ideas on that issue? Thanks

LOCAL_PATH = '/backups/'

parser = OptionParser(description='Specify directory or files')
parser.add_option('-d', '--directory', action='store', dest='directory',
        default=LOCAL_PATH,
        help='specify directory'
)
parser.add_option('-f', '--file',
    action='store', dest='filename',
    default=None,
    help='specify file'
)
#parser.add_option('--version', action='version', version='%(prog)s 1.0')
(options, args) = parser.parse_args(sys.argv[1:])

# just guessing.  Might not be what you want
os.chdir( options.directory)

filenames_or_wildcards = args # take all filenames passed in the command line

# This is just an example and seems rather useless,
# I assume I misunderstood what you wanted
if options.filename is not None:
    filenames_or_wildcards.append(options.filename)

if len(filenames_or_wildcards) == 0:
    filenames_or_wildcards = '*'

# Now expand all wildcards
# glob.glob transforms a filename or a wildcard in a list of all matches
# a non existing filename will be 'dropped'
all_files = []
for filename_or_wildcard in filenames_or_wildcards:
    all_files.extend( glob.glob(filename_or_wildcard) )

# List files in directory and upload them to bucket
#for filename in os.listdir(LOCAL_PATH):
for filename in all_files:
  #skip all directory entries which are not a file
        if not os.path.isfile(filename):
              continue
        # alternative could be to skip only directories and to tread
        #  symbolic links and named pipes and devices.
        #if os.path.isdir(filename):
         #     continue
    DO_STUFF

Open in new window

0
 
LVL 16

Expert Comment

by:gelonida
ID: 35165665
I assume, the problem is some misunderstanding betweenwhat you try to achieve and what I understood.


you can add following lines (for debugging and run the script again)
before line 2  of original script: (the one in your most recent reply):
print "ARGS", sys.argv

after line 7 of original script: (the one in your most recent reply)
print "changing to ", options.directory

after line 27 of the original script (the one in your most recent reply)
print "will check for", filenames_or_wildcards,"in directory",os.getcwd()

after line 34 of original script
print "all files", all_files



0
 
LVL 16

Expert Comment

by:gelonida
ID: 35165749
Here the complete code with debug print statements.
#!/usr/bin/env python
import os, sys, glob

from optparse import OptionParser

LOCAL_PATH = '/backups'

print "ARGS", sys.argv
parser = OptionParser(description='Specify directory or files')
parser.add_option('-d', '--directory', action='store', dest='directory',
        default=LOCAL_PATH,
        help='specify directory'
)
parser.add_option('-f', '--file',
    action='store', dest='filename',
    default=None,
    help='specify file'
)
#parser.add_option('--version', action='version', version='%(prog)s 1.0')
(options, args) = parser.parse_args(sys.argv[1:])

# just guessing.  Might not be what you want
print "changing to ", options.directory
os.chdir( options.directory)

filenames_or_wildcards = args # take all filenames passed in the command line

# This is just an example and seems rather useless,
# I assume I misunderstood what you wanted
if options.filename is not None:
    filenames_or_wildcards.append(options.filename)

print "will check for", filenames_or_wildcards,"in directory",os.getcwd()

if len(filenames_or_wildcards) == 0:
    filenames_or_wildcards = '*'

# Now expand all wildcards
# glob.glob transforms a filename or a wildcard in a list of all matches
# a non existing filename will be 'dropped'
all_files = []
for filename_or_wildcard in filenames_or_wildcards:
    all_files.extend( glob.glob(filename_or_wildcard) )

print "all files", all_files

# List files in directory and upload them to bucket
#for filename in os.listdir(LOCAL_PATH):
for filename in all_files:
  #skip all directory entries which are not a file
    if not os.path.isfile(filename):
        continue
    # alternative could be to skip only directories and to tread
    #  symbolic links and named pipes and devices.
    #if os.path.isdir(filename):
    #    continue
    print "dostuff with %s in %s" % (filename, os.getcwd())

Open in new window

0
 
LVL 16

Expert Comment

by:gelonida
ID: 35165836
Concerning --version

I never heard of the action "version" so I can't comment on it.


There's multiple ways of achieving to display the version.
Once more you must know what you like to achieve.

Shall the version be printed out and the command continue or shall the version be printed out and the command abort.


One sample implementation: (prints version and aborts)

parser.add_option('-v', '--version', action="store_true", dest="show_version",
    default=False, help='displays the version number')

#parser.add_option('--version', action='version', version='%(prog)s 1.0')
(options, args) = parser.parse_args(sys.argv[1:])

if options.show_version:
    prog = os.path.basename(sys.argv[0])
    version_str = "1.0"
    print "version is: %s %s" % (prog, version_str)
    sys.exit(0)

Open in new window

0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 3

Author Comment

by:adamshields
ID: 35165953
Let me work on the first bug. So the problem for using -f, the script seems to be prepending the LOCAL_PATH to the specified file.

$ python boto-backup.py -f /home/nikto-2.1.4.tar.bz2
ARGS ['boto-backup.py', '-f', '/home/nikto-2.1.4.tar.bz2']
changing to  /backups/
all files ['/home/nikto-2.1.4.tar.bz2']
Traceback (most recent call last):
  File "boto-backup.py", line 79, in <module>
    k.set_contents_from_filename(LOCAL_PATH + filename, cb=percent_cb, num_cb=10)
  File "/usr/local/lib/python2.6/dist-packages/boto/s3/key.py", line 675, in set_contents_from_filename
    fp = open(filename, 'rb')
IOError: [Errno 2] No such file or directory: '/backups//home/nikto-2.1.4.tar.bz2'

Open in new window


or if I do not specify the absolute path it also fails

$ python boto-backup.py -f nikto-2.1.4.tar.bz2
ARGS ['boto-backup.py', '-f', 'nikto-2.1.4.tar.bz2']
changing to  /backups/
all files []

Open in new window

0
 
LVL 16

Expert Comment

by:gelonida
ID: 35166086
It does not fail. It just doesn't do what you want ;-)

Loot at line 22
# just guessing.  Might not be what you want
print "changing to ", options.directory
os.chdir( options.directory)



So it seems the behaviour, that you like to have is:

- call without parameters: work on all files of local path

- call with option -d call with all files in directory d

- call with option -f call with all files in the current working directory, that match the ;'file' parameter

- call without options but with multiple args: work on all files matching any of the passed parameters in the current working directory.

The open question would be what you expect if you mix the switches -f -d


The program can be changed but it is important to know what exactly you want to achieve.

Can you please specify?

I'd like to avoid sending several further solutions which will not do what you need.





0
 
LVL 3

Author Comment

by:adamshields
ID: 35166159
- call without parameters: work on all files of local path

- call with option -d call with all files in directory d

- call with option -f call with all files in the current working directory or absolute specified path OR files that match the wildcard ;'file' parameter

- call with -f -d display options, i.e help with usage.
0
 
LVL 16

Accepted Solution

by:
gelonida earned 500 total points
ID: 35166251
This might be what you're looking for.

It just doesn't print out the help text if you pass both arguments.
#!/usr/bin/env python
import os, sys, glob

from optparse import OptionParser

LOCAL_PATH = '/backups'

print "ARGS", sys.argv
parser = OptionParser(description='Specify directory or files')
parser.add_option('-d', '--directory', action='store', dest='directory',
        default=None,
        help='specify directory'
)
parser.add_option('-f', '--file',
    action='store', dest='filename',
    default=None,
    help='specify file'
)

parser.add_option('-v', '--version', action="store_true", dest="show_version",
    default=False, help='displays the version number')

(options, args) = parser.parse_args(sys.argv[1:])

if options.show_version:
    prog = os.path.basename(sys.argv[0])
    version_str = "1.0"
    print "version is: %s %s" % (prog, version_str)
    sys.exit(0)

filenames_or_wildcards = []

# you could abort if two options were specified.
# on the other hand: the script works well even if you add both options
# remove the next lines if you don't need them
if options.directory is not None and options.filename is not None:
    print "invalid combination of command line arguments"
    sys.exit(0)

# remove next line if you do not want allow to run the script without the -f -d 
# option, but with arguments
filenames_or_wildcards = args # take all filenames passed in the command line

# if -f was specified add them (in current working directory)
if options.filename is not None:
    filenames_or_wildcards.append(options.filename)

# if nothing was specified, then use localpath
if (options.directory is None) and len(filenames_or_wildcards) == 0:
    options.directory = LOCAL_PATH

# if -d was specified or nothing at all add files from dir
if options.directory is not None:
    filenames_or_wildcards.append( os.path.join(options.directory, "*") )

# Now expand all wildcards
# glob.glob transforms a filename or a wildcard in a list of all matches
# a non existing filename will be 'dropped'
all_files = []
for filename_or_wildcard in filenames_or_wildcards:
    all_files.extend( glob.glob(filename_or_wildcard) )

print "all files", all_files

# List files in directory and upload them to bucket
#for filename in os.listdir(LOCAL_PATH):
for filename in all_files:
  #skip all directory entries which are not a file
    if not os.path.isfile(filename):
        continue
    # alternative could be to skip only directories and to tread
    #  symbolic links and named pipes and devices.
    #if os.path.isdir(filename):
    #    continue
    print "dostuff with %s in %s" % (filename, os.getcwd())

Open in new window

0
 
LVL 3

Author Comment

by:adamshields
ID: 35166626
Wow that's great, just one last bit, now the "filename" variable includes the path. Is there a simple way to strip the path from the filename?

# List files in directory and upload them to bucket
for filename in all_files:
  #skip all directory entries which are not a file
        if not os.path.isfile(filename):
              continue
        # alternative could be to skip only directories and to tread
        #  symbolic links and named pipes and devices.
        #if os.path.isdir(filename):
         #     continue
        k = Key(bucket)
        k.key = filename
        k.set_contents_from_filename(filename, cb=percent_cb, num_cb=10)

Open in new window

0
 
LVL 3

Author Closing Comment

by:adamshields
ID: 35167241
Disregard my last question, it can be done with: os.path.basename(filename)

Thanks a lot of the help.
0
 
LVL 3

Author Comment

by:adamshields
ID: 35167809
If I add os.path.basename(filename) to the last bit of code everything works except LOCAL_PATH, i.e. not specifying an argument? Any tips?
0
 
LVL 16

Expert Comment

by:gelonida
ID: 35175257
Hmm strange when I tested the script worked without any argument.

Could you please send teh complete script to be sure, that there is no typo ar something like that?
0
 
LVL 3

Author Comment

by:adamshields
ID: 35175270
I think it was an issue on my end, all is working well, thanks for the follow-up.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Suggested Solutions

Flask is a microframework for Python based on Werkzeug and Jinja 2. This requires you to have a good understanding of Python 2.7. Lets install Flask! To install Flask you can use a python repository for libraries tool called pip. Download this f…
Whether you’re a college noob or a soon-to-be pro, these tips are sure to help you in your journey to becoming a programming ninja and stand out from the crowd.
The viewer will learn how to user default arguments when defining functions. This method of defining functions will be contrasted with the non-default-argument of defining functions.
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now