Solved

Python csv errors using Python 2.3 for code written with Python 2.2

Posted on 2004-08-06
1
831 Views
Last Modified: 2007-12-19
I am a new systems administrator (new to my company and new to the field) trying to get some Python code to work- that the previous systems administrator had written.  He used Linux with Python 2.2, importing a csv module to run his code.  I am now trying to run this code on a Macintosh iBook, running Panther (Mac OS 10.3.4) which comes bundled with its own csv module.  

I am getting the following errors when I try to run his code.  (Apparently it used to work fine when he ran it from his laptop on Linux- using Python 2.2)

This code is supposed to mount users' C drives and search for .pst files ( and other files ) and print them out. (The code looks like it may also search for media files and delete non-work related files).  This project came up because my boss would like to know where all the .pst files are (on the Windows NT network that is mounted to our Solaris servers using Samba) Plus he wanted to be able to use the same tools the other administrator used, but  for some reason, I can't get it to run.  

I realize that without being able to connect to my NT network, it will be impossible for anyone to duplicate my exact errors, but thought that someone may have insight into what I can change in this code to get it up and running.  If nothing else, I would like a script to mount user's personal C: drives to search for .pst folders.  I can work on the extra functionality of this code at another time.  

Here are all of the pieces of code that are supposed to work together:

###pstfinder.py calls killjoy.py to find .pst files

import os
import commands
import sys
import csv
from killjoy import Killjoy, Scanner, Writer

class Mailjoy(Killjoy):
    credfilename = '/tmp/pstfinder.cred'
    outputfile = 'pstfinder.csv'

    def execute:
        pass #FIXME use derived Scanner class
   
class PSTScanner(Scanner):
    badexts = ['.pst']



____________________________________
When I run pstfinder.py I get the following errors:
keroleen:~/Desktop/python/killjoy juliann$ python pstfinder.py
  File "pstfinder.py", line 11
    def execute:
               ^
SyntaxError: invalid syntax

_______________________________________
Here is the second half of the code that works with pstfinder.py
______________________________________



###killjoy.py looks for files (.pst files) and prints them out- plus does other clean up activity on non-work related files


#!/usr/bin/env python

import os
import commands
import sys
import csv # this uses the python 2.2 CSV module
import re
import threading
import Queue
import time
import popen2
import getopt
import stat
from usermapping import usermapping

class Killjoy:
    """Scan PCs for media files.  Obtains a list of clients from the domain controller and uses smbfs to mount the C$ share.  Writes output as a CSV file."""
    workdir = '/tmp/killjoy'
    domain = 'ICI'
    credentials = """username=Administrator\npassword=h0t^^s@nd\ndomain=ICI"""
    credfilename = '/tmp/killjoy.cred'
    dmb = 'ici-dc1'
    outputfile = 'killjoy.csv'
    numscanners = 10

    def __init__(self):
        """Make a working directory for mount points and write out a credentials file for mount.smbfs."""
        if not (os.path.isdir(self.workdir)):
            os.mkdir(self.workdir)
        print "** writing credentials to %s" % self.credfilename
        creds = open(self.credfilename, 'w').write(self.credentials)

    def getmountpoint(self, client):
        return os.path.join(self.workdir, client)

    def rmcreds(self):
        print "** deleting credential file %s" % self.credfilename
        os.unlink(self.credfilename)

    def getclientlist(self):
        print "** getting client list"
        command = 'smbclient -A %s -L //%s' % (self.credfilename, self.dmb)
        print "** running command: %s" % command
        status, output = commands.getstatusoutput(command)
        if status == 0:
            # special case for "NT-BACKUP"
            clients = [ client for client in output.split() if (client[:2] == 'NT' and client != 'NT-BACKUP') ]
            # add ICI3 to search profiles, handled as a special case in Scanner
            clients.append('ICI3')
            return clients
        else:
            print "** no client list:\n%s" % output
            sys.exit(1)

    def listnotscanned(self, clientlist):
        """Andy wants to know which machines were in Chester's list but not known by the DC."""
        chesterslist = usermapping.keys()
        missingpcs = [ pc for pc in chesterslist if pc not in clientlist ]
        return missingpcs

    def mount(self, client, driveletter='C'):
        mountpoint = self.getmountpoint(client)
        if not os.path.isdir(mountpoint):
            os.mkdir(mountpoint)
        print "** mounting \\\\%s\\%s$ on %s" % (client, driveletter, mountpoint)
        command = 'mount -t smbfs -o credentials=%s //%s/%s\$ %s' % (self.credfilename, client, driveletter, mountpoint)
        status, output = commands.getstatusoutput(command)
        if status != 0:
            print "status: %d\noutput: %s" % (status, output)
            return None
        else:
            return mountpoint
       
    def umount(self, client):
        mountpoint = self.getmountpoint(client)
        print "** unmounting %s" % mountpoint
        command = 'umount %s' % mountpoint
        status, output = commands.getstatusoutput(command)
        if status != 0:
            print "** command failed. status: %d\noutput: %s" % (status, output)


    def mountall(self, clientlist):
        mounted = []
        notmounted = []
        for client in clientlist:
            driveletter = 'C'
            if client.lower() == 'ici3':
                driveletter = 'J' # search profiles
            if self.mount(client, driveletter):
                mounted.append(client)
            else:
                notmounted.append(client)

        return mounted, notmounted

    def umountall(self, clientlist):
        for client in clientlist:
            self.umount(client)

    def execute(self, clientlist, delete=False):
        writequeue = Queue.Queue(0)
        scanqueue = Queue.Queue(0)


        Writer(writequeue, self.outputfile, delete).start()
        for i in range(self.numscanners):
            Scanner(scanqueue, writequeue, self.workdir, self.credfilename, delete).start()

        mounted, notmounted = self.mountall(clientlist)

        for client in mounted:
            scanqueue.put(client)

        for client in notmounted:
            writequeue.put([{'filename':'SCAN FAILED','client':client,'date':'N/A'}])

        notscanned = self.listnotscanned(clientlist)
        for client in notscanned:
            writequeue.put([{'filename':'NOT SCANNED','client':client,'date':'N/A'}])

        # clean up
        for i in range(self.numscanners):
            scanqueue.put(None)

        while True:
            time.sleep(5)
            if scanqueue.empty():
                writequeue.put(None)
                self.umountall(mounted)
                self.rmcreds()
                break

class Scanner(threading.Thread):
    # these are compiled into a big regular expression
    innocuous = ['clock\.avi', 'cmos\.ram', 'dna connected\.avi', 'dna searching\.avi', 'sample\.mov', 'findfile\.avi', 'Temporary Internet Files', 'Microsoft Visual Studio', 'RealPlayer', 'WindowsMediaPlayer','Netscape.Users.*Cache']
    #badexts = ['.mp3','.mpg','.mpeg','.wma','.wmv','.mov']
    badexts = ['.pst']
    naughtyfiles = {} # a dict of client:[badfiles]

    def __init__(self, scanqueue, writequeue, workdir, credfilename, delete=False):
        self.__scanqueue = scanqueue
        self.__writequeue = writequeue
        self.credfilename = credfilename
        self.workdir = workdir
        threading.Thread.__init__(self)
        self.delete = delete

        # charge up the innocuous file filter
        reify = [ '.*' + item  for item in self.innocuous ]
        self.ok = re.compile('|'.join(reify), re.IGNORECASE)

    def run(self):
        while True:
            client = self.__scanqueue.get()
            if client is None:
                print "* shutting down thread %s" % self
                break # end of queue
            else:
                foundfiles = self.findnaughtyfiles(client)
                if foundfiles:
                    self.__writequeue.put(foundfiles)
               
    def findnaughtyfiles(self, client):
        mountpoint = os.path.join(self.workdir, client)
        filelist = []
        print "** searching %s" % client
        def _visitfunc(filelist, dirname, names):
            #sys.stdout.write(".")
            #sys.stdout.flush()
            for name in names:
                filename = os.path.join(dirname, name)
                if self.ok.match(filename):
                    # if it's ok, we don't care
                    break
                if os.path.isfile(filename):
                    ext = os.path.splitext(filename.lower())[1]
                    if ext in self.badexts:
                        if self.delete:
                            try:
                                os.unlink(filename)
                                print "Deleted",
                            except OSError:
                                print "unable to delete %s" % filename
                        if client.lower() == 'ici3':
                            driveletter = 'J:' # profile drives
                        else:
                            driveletter = 'C:'
                        mtime = os.stat(filename)[stat.ST_MTIME]
                        filedate = time.strftime('%m/%d/%Y', time.localtime(mtime))
                        filename = filename.replace(mountpoint, driveletter).replace('/','\\')
                  print "%s: %s" % (client, filename)
                        filelist.append({'client':client,'filename':filename,'date':filedate})

        os.path.walk(mountpoint, _visitfunc, filelist)
        print '** finished searching %s' % client
        if not filelist:
            filelist = [{'filename':'CLEAN','date':'N/A','client':client}]
        return filelist

class Writer(threading.Thread):
    from usermapping import usermapping
    def __init__(self, queue, outputfile, delete=False):
        self.__queue = queue
        threading.Thread.__init__(self)
        print "** initializing output file %s" % outputfile
        self.fd = open(outputfile, 'w')
        self.parser = csv.parser()

        if delete:
            self.fd.write(self.parser.join(['The following files were deleted:']) + '\n')
        self.fd.write(self.parser.join(['Name','UID','PC','File','Date']) + '\n')

    def writeoutput(self, foundfiles):
        outcsv = []
        for file in foundfiles:
            pc = file['client'].upper()
       
            try:
                pcdata = usermapping[pc]
            except KeyError:
                # provide blank data if pc is not in index
                pcdata = {}
                for i in [ 'cn','uid']:
                    pcdata.setdefault(i)

            cn = pcdata['cn']
            uid = pcdata['uid']

            outcsv.append(self.parser.join([cn, uid, pc, file['filename'], file['date']]))
       
        self.fd.write('\n'.join(outcsv))
        self.fd.write('\n')
        self.fd.flush()

        #print "** output written to %s" % self.outputfile

    def run(self):
        while True:
            item = self.__queue.get()
            if item is None:
                break # reached end of queue
            foundfiles = item # expand tuple
            if item is None:
                break # reached end of queue
            #print "* Writer: writing output for %s" % client
            self.writeoutput(foundfiles)
        print "* Writer: shutting down"
        self.fd.close()

if __name__ == "__main__":
    usage = "Usage: %s [-r] clientlist\n\t-r: delete found files\n"
    try:
        opts, args = getopt.getopt(sys.argv[1:], 'r')
    except getopt.GetoptError, e:
        print usage
        sys.exit(1)

    delete = False # the default
    for o, v in opts:
        if o == '-r':
            delete = True

    clients = args

    kj = Killjoy()
    if not clients:
        clients = kj.getclientlist()

    kj.execute(clients, delete)
________________________________________

###Here is the (abbreviated) list of users:

###usermapping.py

usermapping  = {'NT1100': {'cn': 'Tara Meadows', 'uid': 'tmeadows'}, 'NT1101': {'cn': 'Chris Boyland', 'uid': 'cboyland'}, 'NT1108': {'cn': 'Agnes Thomas', 'uid': 'agnes'}, 'NT1109': {'cn': 'Keith Lawson', 'uid': 'lawson'}}
 ________________________________________

Here is the error message I get when I try to run killjoy.py:

keroleen:~/mbriggs_work/python/killjoy juliann$ python killjoy.py
** writing credentials to /tmp/killjoy.cred
** getting client list
** running command: smbclient -A /tmp/killjoy.cred -L //ici-dc1
** initializing output file killjoy.csv
Traceback (most recent call last):
  File "killjoy.py", line 270, in ?
    kj.execute(clients, delete)
  File "killjoy.py", line 106, in execute
    Writer(writequeue, self.outputfile, delete).start()
  File "killjoy.py", line 208, in __init__
    self.parser = csv.parser()
AttributeError: 'module' object has no attribute 'parser'

0
Comment
Question by:juli_ann
1 Comment
 
LVL 9

Accepted Solution

by:
rjkimble earned 500 total points
ID: 11737597
The execute method def statement is wrong. It should be:

    def execute():

Or something like that. My guess is that it's intended to have arguments. To be consistent with its ancestor (Killjoy), it should probably look something like this:

    def execute(self, clientlist, delete=False):

It appears to me that the Mailjoy class programming has not yet been completed.

There was a major change in the csv module from 2.2 to 2.3. His code would have to be run using Python 2.2 in order to work. Otherwise, you'll have to rewrite it, and that will probably be a fairly significant task. Fortunately, it's possible to install multiple versions of python on one machine. I verified the difference in the csv modules by comparing the 2.2 and 2.3 versions on one of my Debian machines.

Hope this helps.

.... Bob
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

A set of related code is known to be a Module, it helps us to organize our code logically which is much easier for us to understand and use it. Module is an object with arbitrarily named attributes which can be used in binding and referencing. …
Here I am using Python IDLE(GUI) to write a simple program and save it, so that we can just execute it in future. Because when we write any program and exit from Python then program that we have written will be lost. So for not losing our program we…
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
Learn the basics of modules and packages in Python. Every Python file is a module, ending in the suffix: .py: Modules are a collection of functions and variables.: Packages are a collection of modules.: Module functions and variables are accessed us…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now