Effective way of reading txt file (only new content since last read) using Powershell and TaskScheduler


I'm trying to setup a scheduled task that will read the content of a text file and look for a pattern (error messages) and then send an alert. The scheduled task will run every x mins and will read the same file. However, this script has to "remember" where it left off on the last read as to 1) avoid false positives (errors already reported) and 2) save time by reading only 'new' content.

Hoping that someone out there already has overcome this challenge and can point me in the right direction.

I found this piece of code (http://www.archivum.info/microsoft.public.windows.powershell/2008-04/00134/RE-Read-a-huge-text-file-from-bottom-up.html) that uses size to avoid processing the entire file at every iteration, however not sure if size is a good option here.

You're input is appreciated. Thanks.
$TextFilePath = "C:\myfile.log"
$BytesToRead = 1024

$fs = [System.IO.File]::OpenRead($TextFilePath)
$fs.Position = $fs.Length - $BytesToRead

$sr = New-Object System.IO.StreamReader($fs)

$text = $sr.ReadToEnd()



Open in new window

Who is Participating?
QlemoBatchelor, Developer and EE Topic AdvisorCommented:
The code above is the same as the Unix tail command does. But it will not suffice for your needs, as it just reads a constant amount of bytes from the end. Your choice is to either append the file contents to another log file after processing, and deleting the original one; or to remember the byte position last read (the file size). To do the latter you would need to write that value into another file.
bnditAuthor Commented:
@Qlemo - Sorry for the delay in getting back. I like your idea about remembering the byte position from the last read. However, how do you manage multiple files? For instance, if I'm dealing with more than one log file...how can I keep track of the byte positions for each of the log files? I can use a foreach-object loop to step into each of the log files...but inside this loop I'm having trouble writing the byte size to a different txt file.

# logs.txt contains files names (log1.txt..log2.txt...etc)
$logfiles = get-content c:\logs.txt

$logfiles | foreach-object {

# Question here is...how do I grab the corresponding bytecount text file for log1...I thought about a loop, but I dont to process all of these bytecount text files...I only need the corresponding file

Brent ChallisPrincipal: ITCommented:
One approach to this would be to log the processing of each file and record the size of it at that time.  Processing this as a text file would be complex, but if you create the log entries as objects (a set of functions that I use for this is here: http://powershell.com/cs/media/p/10676.aspx ) means that you can retrieve a log, filter by a log entry attribute such as source and the select the latest entry.If the messafe was the file size you could then use this value.
bnditAuthor Commented:
Thanks for the link and the help.
bnditAuthor Commented:
This answer pointed in the right direction to accomplish my goal.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.