The attached scripts I have work but they take a long time to process. They basically filters the records of a log file for a certain string and writes the records that contain that string to a new file. I know execution is directly affectted by the size of the oringinal log file but Is there anyway to speed this up? I have been told that some methods of reading and\ or writting to a file are faster than others. Is the method that I am using the most efficent? If not how do I improve it?
The files that I am filtering are between 20 to 80 MB taking up to 50 seconds.
import os, time
from datetime import datetime, timedelta
def SpecErrLog(File, dt, err, Duration):
source_file = open(File,"r")
file1 = File + " " + dt.replace(":","_") + " [" + err + "]" + str(Duration)
#file1 = "temptry.txt"
dest_file = open(file1, 'w')
Mach, Dt = File.split()
BeginDay = datetime.strptime("00:00:00.000", '%H:%M:%S.%f')
EndDay = datetime.strptime("23:59:59.999", '%H:%M:%S.%f')
dt = datetime.strptime(dt, '%H:%M:%S.%f')
for line in source_file:
LineTimeStamp = datetime.strptime(arr.strip(), '%H:%M:%S.%f') #timeStamp of sourcefile.
upperLimit= dt + timedelta(minutes=Duration)
lowerLimit= dt - timedelta(minutes=Duration)
if lowerLimit > BeginDay and upperLimit < EndDay: #if all records accurr within the same day.
if lowerLimit < LineTimeStamp < upperLimit:
if __name__ == "__main__":
dt = "09:52:15.710"
err = "54300"
SpecErrLog("H108 01-24-2011", dt, err, 30)