URGENT: Sort by Date , Python Script output

What do you want printed, and in what format?

I want to print time-process-complete and eid.
Sort by time-process-complete and eid.

Do you have any idea how large the input files are?

In other words, can the script read the input with a single read, or should the data be processed one record at a time?

input is large, all together file size greater than 135MB. Above attached code  I tested with these files and taking more than 2 hours to complete. But that's fine for moment.

Two hours seems too much to me (for 135 MB). How many records are expected in the result list (i.e. tens, hundreds, thousands, millions)?

The long processing time could be because of the xml.dom.minidom principal features and also because not of its inefficient implementation in Python. I suggest to use the SAX parser in this case. Try the snippet below. The way of compounding the line with info is because of the sorting. It can be generalized -- the info can be collected to tuples, then sort can be done via supplying your own comparison function, and printing the result can also be customized.

I have tested it only with your sample -- one line in result like (tab separated):

2007-05-22T20:00:00+00:00       766877_765765765_load   .\doc.xml

# The script must work with Python 2.3. This way some
# newer features of Python or newer standard moduled 
# cannot be used. SAX parser used.
 
import glob
import os
import sys
import xml.sax
 
class DocHandler(xml.sax.handler.ContentHandler):
    '''The class for handling the SAX parser events.'''
 
    def __init__(self, lst, fname):
        xml.sax.handler.ContentHandler.__init__(self) # init through the base class code
        self.fname = fname # the processed file name
        self.status = []   # for collecting the characters inside the status element
        self.eid = []      # for collecting the characters inside the eid element
        self.tpc = []      # for collecting the characters inside the time-process-complete element
        self.elem = None   # reference to the current container for collecting the chars
        self.lst = lst     # reference to the list of results (not initialized here)
        
 
    def startElement(self, name, attrs):
 
        if name == 'doc':
            # New doc element found. Erase the previously collected status/eid chars.
            self.status = []
            self.eid = []
            
        elif name == 'status':
            # The status element starts. Let's collect the text into self.status.
            self.elem = self.status
            
        elif name == 'eid':
            # The eid element starts. Let's collect the text into self.eid.
            self.elem = self.eid
            
        elif name == 'time-process-complete':
            # The time-process-complete element starts. Collect into self.tpc.
            self.elem = self.tpc
 
        
    def characters (self, ch):
        # If any container for collecting the text is referred, do collect
        # the character into the referred list.
        if self.elem is not None:
            self.elem.append(ch)
 
 
    def endElement(self, name):
        if name == 'doc':
            # The doc element was closed. Process the collected info.
            # Join the lists of collected characters into a string.
            status = ''.join(self.status)  
            eid = ''.join(self.eid)
            tpc = ''.join(self.tpc)
            
            # Collect the compound string of tpc, eid, and file name.
            if status == u'failed':
                self.lst.append(tpc + u'\t' + eid + u'\t' + self.fname)
                
        elif name == 'status':
            self.elem = None               # stop collecting the chars to the self.status 
        elif name == 'eid':
            self.elem = None               # stop collecting the chars to the self.eid
        elif name == 'time-process-complete':
            self.elem = None               # stop collecting the chars to the self.tpc
            
 
def listOfFailed(fname):
    '''Returns the list of eids of the failed documents.'''
    
    # Initialize the resulting list for the eids
    result = []
    
    # Create parser and collect the eid during parsing. 
    parser = xml.sax.make_parser()         # create the SAX XMLReader object
    docHandler = DocHandler(result, fname) # create the object for handling events
    parser.setContentHandler(docHandler)   # use the handler for the parser
    parser.parse(fname)                    # parse
 
    # Return the collected information.
    return result
    
    
def main():
    myPath = '.'               # default path
    if len(sys.argv) > 1:
        myPath = sys.argv[1]   # explicit path to your xml files
        
    # Get all XML file names from the path and process them.
    # Extract the list of failed docs from all xml files into
    # one resultLst.
    resultLst = []
    for fname in glob.glob(os.path.join(myPath, '*.xml')):
        lst = listOfFailed(fname)   # get the failed eids records (as strings)
        resultLst.extend(lst)
 
    # Sort and pring the resulting list.
    resultLst.sort()
    for line in resultLst:
        print line
                
                
if __name__ == '__main__':
    main()
                

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:
106:
107:

Thanks a lot! but when I'm executing this script against my list of all files( 138MB), it gives following error.

---------------------------------------------------------------------------------------------------------
  File "feedback_failed.py", line 113, in ?
    main()
  File "feedback_failed.py", line 103, in main
    lst = listOfFailed(fname)   # get the failed eids records (as strings)
  File "feedback_failed.py", line 87, in listOfFailed
    parser.parse(fname)                    # parse
  File "lib/python2.3/xml/sax/expatreader.py", line 107, in parse
  File "lib/python2.3/xml/sax/xmlreader.py", line 123, in parse
  File "lib/python2.3/xml/sax/expatreader.py", line 207, in feed
  File "lib/python2.3/xml/sax/expatreader.py", line 303, in end_element
  File "feedback_failed.py", line 67, in endElement
    self.lst.append(tpc + u'\t' + eid + u'\t' + self.fname)
MemoryError
---------------------------------------------------------------------------------------------------------


Earlier with dom , I didn't get such error. Only thing was it's getting more than 2 hours time.


Please let me know best solution.

Thanks again!

Dushan.

I cannot try it with Python 2.3. Try to remove the Unicode prefix on the line 60:

self.lst.append(tpc + '\t' + eid + '\t' + self.fname)

or try
   self.lst.append('%s\t%s\t%s' % (tpc, eid, self.fname))

I'm getting same error for both of above line...

File "feedback_failed.py", line 115, in ?
    main()
  File "feedback_failed.py", line 105, in main
    lst = listOfFailed(fname)   # get the failed eids records (as strings)
  File "feedback_failed.py", line 89, in listOfFailed
    parser.parse(fname)                    # parse
  File "lib/python2.3/xml/sax/expatreader.py", line 107, in parse
  File "lib/python2.3/xml/sax/xmlreader.py", line 123, in parse
  File "lib/python2.3/xml/sax/expatreader.py", line 207, in feed
  File "lib/python2.3/xml/sax/expatreader.py", line 303, in end_element
  File "feedback_failed.py", line 63, in endElement
    tpc = ''.join(self.tpc)
MemoryError

Well, then the opposite approach may be needed. The problem is that based on the XML encoding, the collected characters (i.e. one character strings) could be the Unicode ones. Can you post a simplified example of a XML file that is to be processed by the script and that causes the error? I'll try to install the Python 2.3...

Meanwhile, you can try to use something like
    tpc = u''.join(self.tpc)
instead of
    tpc = ''.join(self.tpc)

(also in other similar cases) and the
    self.lst.append(u'%s\t%s\t%s' % (tpc, eid, self.fname))
instead of
    self.lst.append('%s\t%s\t%s' % (tpc, eid, self.fname))

Well, I have just installed Python 2.3.5 and converted the script to use (blindly, mechanically) the Unicode string literals only -- saved in UTF-8 with Signature, LF ended lines. The same way, the doc.xml was converted. It runs without errors on my Windows machine. I do not have Linux at hands. Try to unzip and run the attached file.

Thanks! but I'm sorry to say that still same error.. :( .. seems memory usage is good in here in sax.. shall we try with dom?

sorry memory usage seems not goood in sax .. shall we try with dom.. because for dom it works fine earlier..

can u provide me how to capture exact error with this code using try, catch ..etc

This is superb! I'm getting results within 15 minutes time now.