Effective way of reading txt file (only new content since last read) using Powershell and TaskScheduler


I'm trying to setup a scheduled task that will read the content of a text file and look for a pattern (error messages) and then send an alert. The scheduled task will run every x mins and will read the same file. However, this script has to "remember" where it left off on the last read as to 1) avoid false positives (errors already reported) and 2) save time by reading only 'new' content.

Hoping that someone out there already has overcome this challenge and can point me in the right direction.

I found this piece of code (http://www.archivum.info/microsoft.public.windows.powershell/2008-04/00134/RE-Read-a-huge-text-file-from-bottom-up.html) that uses size to avoid processing the entire file at every iteration, however not sure if size is a good option here.

You're input is appreciated. Thanks.
$TextFilePath = "C:\myfile.log"
$BytesToRead = 1024

$fs = [System.IO.File]::OpenRead($TextFilePath)
$fs.Position = $fs.Length - $BytesToRead

$sr = New-Object System.IO.StreamReader($fs)

$text = $sr.ReadToEnd()



Open in new window

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
The code above is the same as the Unix tail command does. But it will not suffice for your needs, as it just reads a constant amount of bytes from the end. Your choice is to either append the file contents to another log file after processing, and deleting the original one; or to remember the byte position last read (the file size). To do the latter you would need to write that value into another file.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
bnditAuthor Commented:
@Qlemo - Sorry for the delay in getting back. I like your idea about remembering the byte position from the last read. However, how do you manage multiple files? For instance, if I'm dealing with more than one log file...how can I keep track of the byte positions for each of the log files? I can use a foreach-object loop to step into each of the log files...but inside this loop I'm having trouble writing the byte size to a different txt file.

# logs.txt contains files names (log1.txt..log2.txt...etc)
$logfiles = get-content c:\logs.txt

$logfiles | foreach-object {

# Question here is...how do I grab the corresponding bytecount text file for log1...I thought about a loop, but I dont to process all of these bytecount text files...I only need the corresponding file

Brent ChallisPrincipal: ITCommented:
One approach to this would be to log the processing of each file and record the size of it at that time.  Processing this as a text file would be complex, but if you create the log entries as objects (a set of functions that I use for this is here: http://powershell.com/cs/media/p/10676.aspx ) means that you can retrieve a log, filter by a log entry attribute such as source and the select the latest entry.If the messafe was the file size you could then use this value.
bnditAuthor Commented:
Thanks for the link and the help.
bnditAuthor Commented:
This answer pointed in the right direction to accomplish my goal.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.