Solved

Read xml file, and execute command

Posted on 2015-02-22
8
462 Views
Last Modified: 2015-02-24
In question Q_28621867 I asked for a script to read from a log file an execute commands that queried and output the windows event logs to separate files. That works well. I'd like to take it further, and search each event log for the <EventRecordID> tag and the value in it. Using that value I'd like to execute a command similar to this:
wevtutil qe "Microsoft-Windows-Sysmon/Operational" /q:"*[System[Provider[@Name='Microsoft-Windows-Sysmon'] and EventRecordID > xyz ]]"
The bolded items above need to be replaced by their respective log names, and the number found in the <EventRecordID> tag.
Not all logs will contain data, and some that do have data may not have that tag, those should just be skipped.
PowerShell is out, but VBScript or Batch is fine, as long as wevturil is being called by the Vbs or Bat file.
The goal again to be that instead of writing the log over itself, have it look for the highest number in the event log's (in the EventRecordID tag), and then issue appended query results to the respective log files.
1st run: wevtutil qe "Microsoft-Windows-Sysmon/Operational"
2nd run: wevtutil qe "Microsoft-Windows-Sysmon/Operational" /q:"*[System[Provider[@Name='Microsoft-Windows-Sysmon'] and EventRecordID > 123456 ]]"
Again where 123456 was the highest number found in eventrecordID tag in the sysmon event log from the first run. All runs after CountGood >=1 should use the "second run" query.
-rich
0
Comment
Question by:Rich Rumble
  • 5
  • 3
8 Comments
 
LVL 51

Expert Comment

by:Bill Prew
ID: 40625068
Made some headway on this tonight, more to come...

~bp
0
 
LVL 38

Author Comment

by:Rich Rumble
ID: 40625605
In the interest of saving space, and possibly the search times, each query should probably get it's own log file, and then you'd only have to look in the latest file for the highest ID.
Sysmon-1.xml (200Mb)
Sysmon-2.xml (10Mb)
Sysmon-3.xml (20Mb)
It may make sense to log the name of the latest query that completed...
Microsoft-Windows-Sysmon/Operational, 3, 3, Microsoft-Windows-Sysmon-3.xml
Where ...Sysmon-3.xml is the file that completed last. Then the next time the query is run, it will look in sysmon-3.xml for the highest eventnumberID, since it's guaranteed to have the highest.
-rich
0
 
LVL 51

Accepted Solution

by:
Bill Prew earned 500 total points
ID: 40626748
Okay, did some work on this and think I have a decent start on what you described.  Switched to VBS since this has gotten a bit more complex than the last BAT solution.

A few notes:
Start fresh, running the script once to create the new LOG file, then again to archive
The LOG file now contains the log file name, the archive file name (removing "/"), the two counts, and then the last event ID processed
Empty log files are not archived
Each run saves the last record ID processed in the LOG file, and uses it to pick up next time

' Text file I/O constants
Const ForReading = 1
Const ForWriting = 2
Const ForAppending = 8

' Data structure for event file info stored in LOG file
Class LogInfo
  Public ArchiveFile
  Public CountStarted
  Public CountEnded
  Public LastEvent
End Class

' Define output log file location, and command to execute
strLogFile = "log.txt"

' Define tags that surround the "key" to each event record, so that we can find just new entries
strEventIdStart = "<EventRecordID>"
strEventIdEnd = "</EventRecordID>"

' Define a few needed objects
Set objShell = WScript.CreateObject("WScript.Shell")
Set objFSO = CreateObject("Scripting.FileSystemObject")

' MAIN LOGIC : If log file exists, process it, otherwise just create it
If objFSO.FileExists(strLogFile) Then
   ProcessLog(strLogFile)
Else
   CreateLog(strLogFile)
End If

' Done
Wscript.Quit

' Subroutine to process the log file and archive any new events
Sub ProcessLog(strFile)
   ' Load LOG file contents into a dictionary
   Set dicLogInfo = ReadControl(strFile)

   ' Loop through each event file to process
   For Each strKey In dicLogInfo

      ' Check if any events exist, skip empty event logs
      If RecordCountLog(strKey) > 0 Then

         ' Update the LOG file on disk indicating we are starting this event file
         dicLogInfo.Item(strKey).CountStarted = dicLogInfo.Item(strKey).CountStarted + 1
         WriteControl strFile, dicLogInfo

         ' Open the archive file for this event log, append to it if already exists
         Set objLogFile = objFSO.OpenTextFile(dicLogInfo.Item(strKey).ArchiveFile, ForAppending, True)

         ' Query the event file for any events since the last time we checked
         Set objExec = objShell.Exec("wevtutil qe """ & strKey & """ /q:""*[System [(EventRecordID>" & dicLogInfo.Item(strKey).LastEvent & ")]]""")
         lngLastEvent = -1

         ' Process each new event, adding it to archive file, and saving the last event id we processed
         Do While Not objExec.Stdout.atEndOfStream
            strOutput = objExec.StdOut.ReadLine()
            intEventIdStart = InStr(1, strOutput, strEventIdStart, vbTextCompare)
            If intEventIdStart > 0 Then
               intEventIdEnd = InStr(intEventIdStart, strOutput, strEventIdEnd, vbTextCompare)
               intDataOffset = intEventIdStart + Len(strEventIdStart)
               intDataLength = intEventIdEnd - intDataOffset
               lngLastEvent = CLng(Mid(strOutput, intDataOffset, intDataLength))
            End If
            objLogFile.WriteLine strOutput
         Loop

         ' Done with this event log, close the archive file
         objLogFile.Close
         Set objLogFile = Nothing

         ' Update the LOG file on disk indicating we finished this event file, and the last event id processed
         dicLogInfo.Item(strKey).CountEnded = dicLogInfo.Item(strKey).CountEnded + 1
         If lngLastEvent <> -1 Then
            dicLogInfo.Item(strKey).LastEvent = lngLastEvent
         End If
         WriteControl strFile, dicLogInfo
      Else
         ' Update the LOG file on disk indicating we started and finished this event file (since it's empty nothing to do)
         dicLogInfo.Item(strKey).CountStarted = dicLogInfo.Item(strKey).CountStarted + 1
         dicLogInfo.Item(strKey).CountEnded = dicLogInfo.Item(strKey).CountEnded + 1
         WriteControl strFile, dicLogInfo
      End If

   Next
End Sub

' Function to return the number of entries in a specified event file
Function RecordCountLog(strKey)
   lngCount = 0

   ' Query the even log and extract the count of records in it
   Set objExec = objShell.Exec("wevtutil gli """ & strKey & """")
   Do While Not objExec.Stdout.atEndOfStream
      strOutput = objExec.StdOut.ReadLine()
      If Left(strOutput, Len("numberOfLogRecords:")) = "numberOfLogRecords:" Then
         lngCount = CLng(Replace(strOutput, "numberOfLogRecords: ", ""))
      End If
   Loop

   RecordCountLog = lngCount
End Function

' Subroutine to create the initial LOG file containing all event files
Sub CreateLog(strLog)
   Set objLogFile = objFSO.OpenTextFile(strLog, ForWriting, True)

   Set objExec = objShell.Exec("wevtutil el")
   Do While Not objExec.Stdout.atEndOfStream
      strOutput = objExec.StdOut.ReadLine()
      strOutput = strOutput & "," & Replace(strOutput, "/", "_") & ".txt,0,0,0"
      objLogFile.WriteLine strOutput
   Loop

   objLogFile.Close
   Set objLogFile = Nothing
End Sub

' Function to load the LOG file into a dictionary
Function ReadControl (strFile)
   Set dicLogs = CreateObject("Scripting.Dictionary")

   Set objFile = objFSO.OpenTextFile(strFile, ForReading, False, TriStateUseDefault)
   strData = objFile.ReadAll
   objFile.Close

   arrLine = Split(strData, vbCrLf)

   For Each strLine In arrLine
      If strLine <> "" Then
         set objLogInfo = New LogInfo
         arrField = Split(strLine, ",")
         objLogInfo.ArchiveFile = arrField(1)
         objLogInfo.CountStarted = arrField(2)
         objLogInfo.CountEnded = arrField(3)
         objLogInfo.LastEvent = arrField(4)
         dicLogs.Add arrField(0), objLogInfo
      End If
   Next

   Set ReadControl = dicLogs
End Function

' Subroutine to write out the dictionary to the LOG file
Sub WriteControl (strFile, dicLogInfo)
   Set objFile = objFSO.OpenTextFile(strFile, ForWriting, True)
   strData = ""

   For Each strKey In dicLogInfo
      strData = strData & strKey & "," & dicLogInfo.Item(strKey).ArchiveFile & "," & dicLogInfo.Item(strKey).CountStarted & "," & dicLogInfo.Item(strKey).CountEnded & "," & dicLogInfo.Item(strKey).LastEvent & vbCrLf
   Next

   objFile.Write strData

   objFile.Close
End Sub

Open in new window

~bp
0
 
LVL 38

Author Comment

by:Rich Rumble
ID: 40627003
THIS.IS.AWESOME. I'm going to open yet another question, I'd like to send these log to a central location, and the initial upload is worrisome to the network and storage teams. I'd really like to compress the data before sending, and data that doesn't get sent have it queued to send. If you're interested, I'll post "part 3" in just a bit!
Thanks again, this is a fine piece of work!
-rich
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 38

Author Closing Comment

by:Rich Rumble
ID: 40627005
Super work, and it's VBscript so it makes it a bit easier to get through our red-tape at work!
0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 40627095
Glad that was helpful.  Think through the next piece and work up your ideas.  A few random thoughts that come to mind:

What frequency will this run on each PC?
What compression technology would work, ZIP, CAP, ...?
If you compress then you will probably need a separate file for each day or run of the extract.
What would prevent files from being sent to the server and require a queue manager?
What will these files ultimately be used for?  If they are just in separate files / ZIP files, it will be very hard to search them for specific items down the road.
~bp
0
 
LVL 38

Author Comment

by:Rich Rumble
ID: 40627167
Correct on all counts. They will likely run hourly, compression I'm messing with, CAB is better for these logs than Zip (compressed folder) http://blogs.technet.com/b/heyscriptingguy/archive/2006/07/06/how-can-i-create-a-compressed-folder-and-add-files-to-it.aspx but 7zip is best overall. 7zip isn't part of the normal build, however I have in the past converted binaries to base64 in Vbscript and then output them when needed. So it's possible to deliver an exe in a vbs that way. Your also correct about queuing, in the past we did a simple "did it send completely? if yes, delete the file that is over 7 days old". The beauty is "searchability" isn't even being asked for, this is more of a forensic item rather than what you'd expect with a SIEM or other central logging solution. In any case that's what zgrep is for :) We aren't going to top notch, more of a patchwork solution until a more proper solution can be stood up.
I'm going to experiment with a few things and I should have a new Q for you in 8-12 hours.
I'll cross post the link here. Thanks again!
-rich
0
 
LVL 38

Author Comment

by:Rich Rumble
ID: 40628418
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Introduction: Recently, I got a requirement to zip all files individually with batch file script in Windows OS. I don't know much about scripting, but I searched Google and found a lot of examples and websites to complete my task. Finally, I was ab…
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now