Parse Astaro/Ulogd file using PowerShell?
I posted a question on this already, at https://www.experts-exchange.com/questions/27775849/A-way-to-import-parse-this-Astaro-firewall-log-format-ulogd-into-an-Excel-workbook.html?anchorAnswerId=38141499#a38141499 , and got a solution using Excel VBA.
But I am wondering if there is an easy way to parse this stuff using PowerShell, and then spit it out in CSV format.
See the original question for full details.
But I am wondering if there is an easy way to parse this stuff using PowerShell, and then spit it out in CSV format.
See the original question for full details.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
I didn't expect you to find out yourself how that works ;-). It took me a while myself to figure out a proper RegExp method, which even is dynamic.
In fact, the dynamic part does not work that well. Should the log entry format change within the logfile, the export will not contain the new "columns". But that is a restriction of CSV and similar formats, which usually only check for a certain amount of rows to determine the format.
Most PowerShell cmdlets will not wait for more than the first line; else they could not be used in non-blocking (= streaming) mode. Sort-Object and Group-Object are cmdlets which have to consider the whole stream, and so they wait until all content has been passed, while e.g. export-csv starts as soon as possible. That speeds up processing.
For future readers:
The while loop is using the conincidence that the remainder of the log line, containing key=value pairs, is collected as $matches[3], the 3rd matching expression in the RegExp. So both the starting expression and the loop expression need to match the "tail" to be processed as that 3rd pattern.
In fact, the dynamic part does not work that well. Should the log entry format change within the logfile, the export will not contain the new "columns". But that is a restriction of CSV and similar formats, which usually only check for a certain amount of rows to determine the format.
Most PowerShell cmdlets will not wait for more than the first line; else they could not be used in non-blocking (= streaming) mode. Sort-Object and Group-Object are cmdlets which have to consider the whole stream, and so they wait until all content has been passed, while e.g. export-csv starts as soon as possible. That speeds up processing.
For future readers:
The while loop is using the conincidence that the remainder of the log line, containing key=value pairs, is collected as $matches[3], the 3rd matching expression in the RegExp. So both the starting expression and the loop expression need to match the "tail" to be processed as that 3rd pattern.
ASKER
It took me a little time to understand how that "While ($Matches" loop works. Very sneaky.
Thanks.